A solution to this is to provide repository tarballs to users which contain a snapshot of the repository. Once downloaded and expanded, you have a viable git clone; run a pull and you’re done (it’s pre-configured to fetch new refs from anongit.kde.org). Actually, run the init script and then a pull; to keep the tarball size as small as possible, it doesn’t include a checked-out working tree, only the .git directory and a tiny script that will a) delete itself and b) check out the working tree.

Now, the idea is to be able to start downloading at one time and to finish it later. The key here is that HTTP file transfers are resumable, so if you start downloading the tarball you can pick up where you left off. However, there was a question — how to be able to show a consistent link on Projects without a priori knowledge of tarball characteristics (since we can’t know any from within Redmine). In Redmine, all we could really do, without massive hackery, is display a statically-named tarball — in this case, it always ends in “-latest.tar.gz”. You can see this at https://projects.kde.org/projects/playground/network/aki/repository or any other project’s repository tab — note the “Tarball” checkout method.

This means however that the client needs some way of knowing whether it’s the same tarball or has been regenerated since. An easy way to do this is to use the If-Unmodified-Since header; however, the web server in use currently doesn’t support this (I checked and the author didn’t believe it was widely used; I pointed him to cURL/libcurl). The other problem with this approach is that it’s not very visible to the user.

So, I first changed the script such that the tarballs being generated weren’t actually named *-latest.tar.gz, but instead that they had more descriptive names and had a -latest.tar.gz version symlinked to the actual tarball. I then wrote a simple web service that the web server proxies to whenever it finds a “-latest.tar.gz” file being requested. This service resolves the symlink and redirects the client, so the user sees immediately what the name of the actual file is. For instance, right now a tarball of the aki repository via http://anongit1.kde.org/aki/aki-latest.tar.gz forwards to http://anongit1.kde.org/aki/aki_20101202050239_sha1-5f866b3a42872f8fd54adcf30bcb8a3a79d02542.tar.gz

Note the components of that filename: the name, the date/time stamp (to the second), “sha1″ indicating that that’s the algorithm to check the hash against, and the hash of the file for easy verification of the completed download. This should make it both easy for the client and the user to verify that it’s the same file, in addition to making it easy for the user to verify the integrity of the full file since they can check the sha1sum against the filename itself.

I think this provides a pretty nice solution to the problem.

First, we now have three anongit servers in rotation. We could probably make do with 0.25 of a single anongit server at the current load, but that will change quickly once kdelibs and co switch over from SVN, so it’s good to be ready.

Second, the generated commit URLs will always be given with the full commit hash, but the commit resolver now supports using partial hashes in case you want to shorten it yourself. For instance, you could manually change the commit URL http://commits.kde.org/amarok/cf17de39f9021064a713db965487be6e3d75a186 to http://commits.kde.org/amarok/cf17de39 to give out a shorter URL.

That’s all; if you were in the States I hope you had a good Thanksgiving.

anongit servers

We now have two anongit servers (the second one will be coming online on Monday or so — it’s 95% done). It is very likely that git:// and http:// protocols will be restricted to the anongit servers for better load balancing, so if you have been using git://git.kde.org/… as your clone URLs, please switch those to git://anongit.kde.org/… URLs. Wait, did I just say http?

http protocol support

The anongit servers support pulling/cloning via http. This is using the newer Smart HTTP support in which the Git protocol is run over HTTP via a special server. All pushing must be done via SSH (port 22 or 443) on git.kde.org.

trash/destroy

There are now two systems of deleting your personal clones/scratch repositories.

• unlock/destroy: You can destroy a personal repository, at which point it is gone forever; however, for safety you are now required to run the “unlock” command on the repository first.
• trash: You can trash a personal repository. When you do this, it will be saved for 28 days in case you change your mind. You can use the “list-trash” command in order to see the repositories you have in the trash.

More details can be found in the Git(.kde.org) Manual.

who-pushed

There is now a “who-pushed” command that, given a SHA1 and a repository, can tell you what user pushed that SHA1 to the repository. This is essential for auditing, since commit authorship information can be faked. Now, if a problem arises, we at least can know who introduced that malevolent commit into the public repository.

Repository nicknames

Remember my last post about short Commit URLs? One thing I wanted — and was much requested — was a way to have a more friendly repository identifier. Now they exist. If a friendly identifier exists for a repository, it will automatically be used in the commit URL that is generated. If you want a friendly URL, let the sysadmins know. (We need to figure out a way to make nice names for repo names longer than 8 characters — or just remove that artificial restriction.)

Clones path change

When we first started out, clones would live somewhere like clones/amarok.git/mitchell/myamarok. Why the “.git”in the middle  (we were asked a lot)? Because the path was meant to match the actual name of the repository as it exists on the server.

The reason for *that* is that Git uses a “.git” suffix on repositories that are bare, to make it clear that it’s a Git directory. However (as we found out from Sitaram), this is not supposed to be user-facing, which is why if you use a Git URL with “.git” at the end of it, it’s dropped in the folder that actually is created on the client. So either the path has to lie about the original path on the server, or the client/user has to be made aware of it when it doesn’t really need to be.

What we ended up doing is removing that “.git” from the middle of the repo name (so now it’d be clones/amarok/mitchell/myamarok) and also took pains to ensure that when clone URLs are shown to the user, they are shown without a trailing “.git”.

At the same time, Sitaram made changes in Gitolite to better handle (in the various commands and in the built-in functionalities) cases where the user uses does or does not use the “.git” suffix so that it works correctly in each case.

Updated permissions on master repositories

Master repositories now allow users to create new branches; however, deletion requires the chosen repository manager(s) to intervene.

Updated permission granularities on personal repositories

Personal (clones/scratch) repositories now have a greater level of permission control settable by the user. (The git.kde.org user manual is not yet updated with this information.)

These are now the permission levels that the user can set, from least to most:

• All: everyone has read and basic write permissions on each repository
• Writers: writers can write new branches to the repository
• Managers: managers can also delete branches from the repository
• Dangers: people designated as dangers (to the repository!) can also rewind/force push to the repository
• Creator: the creator always has full rights

projects.kde.org tree view

If you check out the projects page on projects.kde.org (here) you’ll see that it now supports a tree view matching the structure of the KDE repositories. The URLs also match this structure.

projects.kde.org last activity indicator

On the same page (here) you’ll see that information about the last commit for that project is now shown next to each project.

…aaaaaaaaand we’re done.

Redmine powers Projects at http://projects.kde.org and is intended (pending theming and greater population of data) to be the real “home” of all of KDE’s projects, putting project information, news, repository browsing and more in a pleasant UI. Each project with a git repository will have a project on Redmine as well as on ReviewBoard.

However, Redmine doesn’t have *all* of the git repositories available. This is because in the short term at least there won’t be projects created for user clones of repositories. In addition, the “scratch” area, where KDE developers can maintain their own repositories for anything they wish (that’s KDE-related of course) will never be put in Redmine.

(The reason for this is that if a developer just wants to play around with some code, or perhaps wants a location to store their (versioned) emacs/vi config files or some such thing, there’s no reason that needs to be an “official project” with an entry in Redmine and ReviewBoard. Similarly, if a developer is writing new code but feels that the code is far too raw to actually have others having eyes on it, we believe it should be up to them when they decide to have it put into a project on Redmine and ReviewBoard.)

So, let’s say you’ve just pushed some code. Where would you go to see it on the Web — Redmine or gitweb? I took a first cut at solving this problem by, based on the location of the repository you pushed to, spitting out a Redmine- or gitweb-based URL in the output sent back to your git client. Unfortunately, these were quite long. For example:

http://projects.kde.org/projects/repo-management/repository/revisions/dcd43aacaa806a7a32779a0215b7ab8ed7b05dc8

This isn’t so nice. Especially if your terminal is only 80 characters wide.

So, I came up with a solution — commits.kde.org. It’s a simple Sinatra/Thin-based web application that parses a URL generated by the gitolite hook and forwards you to the correct place. If the repository is on Redmine, it forwards you there; otherwise it forwards you to gitweb.

These aren’t tiny URLs in the style of bit.ly, but they’re deterministic and not based on a database backend. In fact, you can construct these on your own and they’ll still resolve. The form is

http://commits.kde.org/<repoid>/<commitid>

The repoid can be found by looking in the URL that is spit out when you push your code. It stays fixed for each repository (and if it does change, aliases can be added to keep old URLs alive).

In this format, the above URL goes down to

http://commits.kde.org/99c5fdd6/dcd43aacaa806a7a32779a0215b7ab8ed7b05dc8

By doing this, the URL size drops from 110 characters-ish (depending on the name of the project, whether it’s on gitweb or Redmine, and so on) to a fixed 72; enough to make it fit on a single line in most terminals (with the “remote: ” prepended to the git output it’s 80 characters total), and to make it relatively tweet-/dent-able if you don’t need to include much other information in the post.

Update: I should mention that you can shorten those URLs further, but how much depends on when you start getting a collision. If gitweb encounters an ambiguous commit ID, it will 404, without giving information as to why it returned a 404. Redmine, however, will simply return one or the other of the commits — so you may get the wrong commit shown without even realizing it. Worse, this could also mean that URLs that were shorter (very short) but once worked may not work later if a collision comes up later. So you can use as many or few of the hash characters as you want, but I’d stick with at least 8 or so for safety. *Also*, right now the webapp only matches full hash values in Redmine’s database, so if you shorten it you will always get a gitweb URL.

For anyone interested, the current webapp code is GPLv2+ and can be found right here.

Some people can’t be helped.

Today I saw a post by The Linux Critic. In the post the author, Trent, says “I’m really not a fan of the new Amarok“, which naturally got me curious as to why. After all, if you don’t know what problems people have with your software, it can be hard to improve it.

I had never interacted with Trent or any other member of The Linux Critic before, nor had I ever visited their site. I was there with the best of intentions.

I’m going to let the discussion we had speak for itself. I’ve reproduced it below, because Trent started removing my comments and modifying his own. You can see the original blog post here — it’s not even really relevant to most of the ensuing discussion — however I strongly recommend, in case further modification has been made to the discussion on that page, that you read the reproduction of the discussion below which is accurate as of the time of writing this post. At the beginning Trent was fairly hostile; by the end he was outright nasty.

Me:

What about Amarok 2.3 are you not a fan of?

Trent:

The fact that now it’s just another iTunes clone.

Me:

Um. Right.

I guess I will just assume you’re mixing Amarok up with Banshee or Rhythmbox. No other explanation for it.

Trent:

*sigh*

The purpose of this discussion wasn’t to discuss the problems with newer versions of Amarok. It was to discuss the new music player, Clementine

I guess I will just assume you’re mixing Amarok up with Banshee or Rhythmbox. No other explanation for it.

No. I’ve tried Amarok 2.

I don’t like the layout.

I don’t like the entire UI.

I don’t like that the playlist is too small and in the wrong place.

I don’t like the volume control.

I don’t like that it requires KDE4 to run.

I don’t like the enormous waste of space that is the center panel.

I don’t like the ugly look of the buttons.

I don’t like that I can’t figure out how to do a lot of the things I do in Amarok 1.4.

If it were more like Amarok 1, I’d probably be fine with it.

Me:

Sure, I understand the purpose was to discuss Clementine, but when you said you weren’t a fan of the new Amarok, I was interested in knowing why.

The reasons you give are at least mostly valid — aesthetic differences, which are personal for everyone — as opposed to calling it an iTunes clone, which would be very hard to justify. (I say mostly because it doesn’t require you running KDE4 to run. And if you take issue with it depending on KDE libraries, then it seems odd that this would not be an issue for you with Amarok 1.4, which also depended on KDE libraries.)

Trent:

Amarok 1.4 depended on KDE3 libraries. I’m not opposed to KDE3.

Me:

I don’t know why you’re opposed to KDE4. But it doesn’t really matter; they’re just libraries. I don’t run GNOME, but I don’t get upset when something depends on GTK+. If it’s a useful application, then it’s a useful application.

I then attempted to be helpful, by letting him know that many of his complaints could be addressed if he wished to give Amarok another try:

Me:

FWIW — again, not to go into too much detail, since as you said this post is about Clementine — you can adjust Amarok 2′s entire layout, as well as its playlist layout, however you want.

If you go to the View menu you can unselect the Context View if you don’t want it. You can also unselect Lock Layout and relayout the entire thing — have parts of it top-to-bottom, change the order…you can even drag the components on each other to turn them into tabs, so that your entire view could be the playlist, except for when you want to switch to the Media Sources pane to add music to it.

If you go to the Playlist menu, you can select one of four pre-defined playlist layouts, or customize it to your own layout, including a single line per track with whatever data shown that you want.

Also under View, you may want to check out the Slim toolbar — it has a different volume slider.

This is all based on Amarok 2.3.1; not sure which version(s) you’ve tried.

Trent:

Amarok 2.2 was the last one I tried. One could change a lot of that there too, but after about an hour of futzing around with it I just found myself getting too frustrated with it.

I shouldn’t have to go to those lengths just to get around the goofy design philosophy that went into Amarok 2 (much like the rest of KDE4 for that matter).

It’s like they took a look at Amarok 1.4 and said “Ok, let’s see… how can we break everything about this?”.

Ugh.

Clementine isn’t even close to finished yet, and it’s already more usable out of the box than Amarok 2.

Unless they made some rather massive, sweeping usability changes from 2.2 to 2.3.1, which I doubt.

Me:

I don’t remember all the changes between 2.2 and 2.3, but there are plenty. The Amarok team has always been very responsive to user feedback, and that has driven much of our development in the Amarok 2 series just as it did with Amarok 1.

Regardless of you considering the design philosophy of Amarok 2 to be goofy, the entire point of Amarok was always to provide contextual information to your music — Rediscover Your Music. Amarok 2′s goal was to take that further, by allowing for a context space that didn’t have to compete with your collection browsing, so you can still e.g. see lyrics to the song that’s playing while you browse your collection for the next track.

These days Amarok 2 is pretty much feature-complete w.r.t. 1.4 features, and has huge numbers of features never in Amarok 1. It’s always interesting to me how if you look at Amarok 1.1 vs. 1.4 you see *huge* differences, and people tried out each release and gave feedback which made things even better. With Amarok 2 some people seem to think that it’s static, and never changing, and never improving — so they, say, try Amarok 2.1, don’t like how it works or miss some feature that hadn’t been ported yet, and don’t bother trying 2.3, regardless of the fact that in the same series of point releases in the Amarok 1 days they would have seen massive changes. So instead of providing useful feedback and helping Amarok grow and improve, which it still continues to do quite rapidly, they simply turn venemous about it, often complaining about things that have long been fixed or capabilities that have long since been added.

Fortunately, not everyone is like that, and we still have a great user base that provides feedback that we try to address. For instance, the ability to change layouts and make the playlist hugely customizable was the direct result of feedback from Amarok 2 users that wanted to be able to make it behave similarly to Amarok 1.4, which it can do to a very large extent.

Trent:

So instead of providing useful feedback and helping Amarok grow and improve, which it still continues to do quite rapidly, they simply turn venemous about it, often complaining about things that have long been fixed or capabilities that have long since been added.

I’ve seen the KDE forums and I’ve seen enough of the attacks on users who try to provide feedback. It was enough to tell me that if a person isn’t gushing with love over it, a person is therefore just “resistant to change” or “doesn’t get it”.

Most of the things I label as bugs or design flaws are pointed out time and time again as intentional in KDE4 and related applications.

The philosophy itself I find repugnant. That’s why I left KDE completely after trying KDE 4.0, 4.1, 4.2, and 4.3.

While I’m sure Amarok 2.3.1 might be different and maybe even flexible enough to emulate Amarok 1.4, I’ve found that every time I’ve tried anything KDE related in the last 2 years I get frustrated, angry, and venomous.

So no, I haven’t been forthcoming with feedback. I never even know where to begin. There are so many things wrong with the directions KDE4 and Amarok devs have gone, I find myself overwhelmed any time I try to enumerate them.

And again, this discussion is about Clementine, not Amarok 2.3.1. Amarok 2.3.1 might be awesome, that’s great. For people that don’t mind the layout, and the difficult UI, and all the other stuff I simply can’t stand, that’s fine by me. They can use it.

But as Jules said in Pulp Fiction, “Sewer rat might taste like pumpkin pie, but I wouldn’t know, because I wouldn’t eat the filthy m**********r.”

Fortunately, not everyone is like that, and we still have a great user base that provides feedback that we try to address.

Ah. Gotcha. “We”. Meaning that you’re one of the Amarok developers that ruined my favorite music player.

Well, that explains why you’re trolling my blog post.

Me:

Yes. Trolling. Asking for details as to a comment you made in your blog post, and then trying to provide helpful information based on your feedback. Right.

I guess I shouldn’t be surprised. From your comment, you clearly are determined to put down KDE4 and anything related to it, and attempts to be helpful are unwanted.

This is the point where Trent starts modifying history. This was his original reply:

Considering that I don’t use KDE4, I’m not sure that “attempts to be helpful” are even relevant.

To which I replied:

You listed complaints about Amarok 2; I explained how they could be addressed. That’s attempting to be helpful, and relevant to the discussion.

At this point Trent deleted my comment above, and went through two revisions of his previous comment. The first one is below:

Considering that I don’t use KDE4, I’m not sure that “attempts to be helpful” are even relevant.

As an FYI to any other KDE developers or fans that might be reading: If you have something useful to contribute to any discussion on The Linux Critic, your comments are welcome.

If all you are going to do is pick fights about why EVERYBODY can’t love every little thing KDE4 and its apps, your comments are not.

This has happened again and again, and I’m not going to tolerate it any further.

And here is his second revision, which is still the latest at the time of publishing this post:

Considering that I don’t use KDE4, I’m not sure that “attempts to be helpful” are even relevant.

As an FYI to any other KDE developers or fans that might be reading: If you have something useful to contribute to any discussion on The Linux Critic, your comments are welcome.

If all you are going to do is pick fights about why EVERYBODY can’t love every little thing about KDE4 and its apps, your comments are not.

This has happened again and again, and I’m not going to tolerate it any further. Play nice, or play somewhere else.

Thanks.

So there you have it. Not much else to say. I’d never heard of The Linux Critic before, and now I know why.

Update: Trent replied with more accusations of trolling (does he even know what that word means, considering he’s the one trolling on KDE4?), of purposefully trying to derail a discussion about Clementine (what discussion?), of some sort of KDE4 lovefest, and all sorts of other manners of odd accusations against a person he’d never met before today and the discussion above. This guy is off his rocker. Here’s the comment:

And I see my KDE troll has gone off to sulk.

Just as a heads-up, trying to derail a discussion about the Clementine music player and turning it into all about KDE4 is not “trying to be helpful”, and it is not “relevant”. It’s rude, insulting, and I consider it to be trolling.

While I don’t typically behave like a topic nazi here with respect to keeping every single comment on-topic, when it comes to KDE4 trolls, I have very little patience, because those comments always end up going the same places, and it’s never constructive, always just fight-picking.

In this case, I’ll freely admit at least some culpability, because I responded to the initial question about Amarok, despite my better judgement.

Here’s a hint: not everybody loves KDE4. Not everybody wants to use it. Some of us prefer not to use KDE4, and some of us (gasp!) even prefer the KDE3 versions of those apps!

Baffling, I know. But it’s true. That’s why I made this post yesterday about Clementine. There are lots of other people out there that used to love Amarok 1, and don’t want Amarok 2.

For those people, seeing a new, stable, usable fork of Amarok 1.4 is a really great and exciting thing, and I thought that others should know about it. It’s very positive news when for a lot of us, the past few years have been spent trying in vain to find a replacement for that now dead application.

And in my experience, when KDE4 trolls come around, trying to change the discussion into an argument about why????????, it’s not relevant, not helpful, and is anything but useful to the topic at hand.

The reason I’m commenting like this is because I want to clarify that. I’m drawing a line on this subject because I’m tired of this.

To Jeff: those of us who have left the KDE world don’t want your help. You can keep it. Don’t call us. We’ll call you. So scurry off to your KDE4 lovefest and leave the rest of us be to solve the problems you created for us. You’ve done more than enough.

What a swell guy.

Update 2: Here’s Trent’s response to one person who attempted to go onto his blog and tell it like it is (thanks Marand!). One editorial note: Trent claims “you also didn’t see some of the other garbage of his I deleted.” Every tiny bit of my interaction with him is in the original post above. Not one word has been omitted. It’s possible he has since deleted some more of my comments from his blog in order to rewrite history, as I have shown above, but what you see above in the original post is all of the “garbage”.

Marand:

Maybe I’m missing something, but I don’t see how he was trolling. I see a developer trying to communicate with a former user to get useful feedback and being treated with hostility for it.

If you don’t like something, that’s fine. If you can’t explain why you don’t like it, that’s also fine. However, it isn’t his fault you can’t offer a well-reasoned explanation of why you dislike the application, so I think you crossed a line lashing out at him over your inability to express your opinions constructively.

It looks like the discussion fell apart when constructive advice was offered to solve your various complaints. Lacking better arguments, you lashed out. You should have just left it at “I don’t like it because it’s different, and I have an irrational hatred of the number 4″, it would have been more mature.

Here, let me give you a better example to use when saying you dislike Amarok 2. This is a problem that has plagued the app for me in all versions since the change. Frequently (several times a day), I have to restart Amarok 2 because the dynamic playlist breaks and stops updating. Haven’t been able to figure out why, but it seems to be related to the collection scanning. It’s frustrated me enough that I’ve considered changing players, though I like Amarok enough (even 2!) to still use it despite this problem.

For the record, I’m not a developer of KDE, Amarok, or anything else. I use KDE4 and I actually like it, but I don’t care what anyone else uses or likes. My only motivation for this post is I think that, if you like or dislike something, you should either be able to provide clear explanation of why, or you should admit you cannot.

Trent:

Maybe I’m missing something, but I don’t see how he was trolling.

He was, as happens frequently, someone who tried (successfully, I might add) to derail a topic to talk about something else after being repeatedly warned that it was off-topic. While that can pretty easily fit the definition of someone needing their comments moderated, he was doing it in an inflammatory, extraneous manner. You also didn’t see some of the other garbage of his I deleted.

My biggest mistake was in responding to him to begin with, given that his very first comment was off-topic (and I know from experience where those usually end up leading).

If you don’t like something, that’s fine. If you can’t explain why you don’t like it, that’s also fine. However, it isn’t his fault you can’t offer a well-reasoned explanation of why you dislike the application, so I think you crossed a line lashing out at him over your inability to express your opinions constructively.

I’m perfectly capable of explaining why I don’t like Amarok 2. If you bothered reading before you posted, you’d notice that I gave him several valid reasons why I don’t use Amarok 2, despite the fact that this article was about something else, specifically a program that Amarok 1.4 users may not be aware of that I’ve found to be a very positive direction in music player development for a change.

And considering that The Linux Critic is my blog — not his, and not yours –, and it’s my prerogative to determine content here, including content of discussions about posts, and considering that his discussion was off-topic and inflammatory, I was in no way “out of line” in lashing out at him. As I’m sure you know, Jeff has his own blog where he went to whine, and note that I didn’t go over there and change the subject to something that he found inappropriate, or spout idiocy at his opinions. He can have his opinions, he can say what he wants.

And he can express them somewhere else. They were inappropriate here. As I mentioned, I made a mistake in responding to him to begin with.

It looks like the discussion fell apart when constructive advice was offered to solve your various complaints.

The discussion “fell apart” when he tried to change the subject from Clementine 0.4 to Amarok 2 and KDE4. After repeated warnings he continued to do it, so I deleted his further attempts at doing so and I blacklisted his IP. I have made it very clear here in the past that I will not tolerate spammers and trolls, and I felt that I was pretty reasonable in the amount of leeway I gave him. Most of the time I just delete those types of comments outright, because they inevitably lead to (surprise!) more off-topic discussion like we’re having right now. 

Lacking better arguments, you lashed out.

Ah. So, since I went out of my way to present arguments that were in response to off-topic, inflammatory trolling, and you happen to disagree with my opinion, they’re “lacking better arguments”. Gotcha.

You should have just left it at “I don’t like it because it’s different, and I have an irrational hatred of the number 4″, it would have been more mature.

And you were wondering why I don’t like KDE4 trolls commenting on my blog posts? Again, you apparently missed the several valid reasons I gave why I dislike Amarok 2. And you apparently missed the part where I mentioned that I’ve used KDE 4.0, 4.1, 4.2, and 4.3, and have dismissed it as a valid replacement for KDE 3.5.10.

If I didn’t like things because they are “different”, I’d still be using Windows 2000.

And on the subject of maturity, derailing a topic to insult people because you disagree with their opinions, that wins a prize too, sport. Keep it up.

Here, let me give you a better example to use when saying you dislike Amarok 2. This is a problem that has plagued the app for me in all versions since the change. Frequently (several times a day), I have to restart Amarok 2 because the dynamic playlist breaks and stops updating. Haven’t been able to figure out why, but it seems to be related to the collection scanning. It’s frustrated me enough that I’ve considered changing players, though I like Amarok enough (even 2!) to still use it despite this problem.

And the next time I write an article here about the problems with Amarok 2 and how they could be improved, that will be a valid and relevant thing to discuss. But considering that this particular one is about Clementine, that would be a bit out of place here, as I told Jeff several times before I banned him.

I use KDE4 and I actually like it, but I don’t care what anyone else uses or likes.

No, like other KDE4 fanboys I’ve encountered, you obviously can’t stand it when someone doesn’t get all starry-eyed at the desktop environment you love so much, and just have to shoot your mouth off any time someone expresses an opinion to the contrary, whether it’s appropriate for you to do so or not.

My only motivation for this post is I think that, if you like or dislike something, you should either be able to provide clear explanation of why, or you should admit you cannot.

And I did, above, in response to Jeff. Even though it was against my better judgement, and in violation of my own policy against feeding the trolls here as it was off-topic and inflammatory.

What I probably should have done was delete his posts and ban him right off the bat. Had I known he was just an Amarok developer coming here to pick a fight, that’s what I would have done.

The biggest problem is, I tend to give people far too much leeway here sometimes, and look what it gets me.

Yes, look what it gets him.

Update 3: Another brilliant post from Trent in the comments of his blog. The real highlight is his characterization of my interaction with him, in which I apparently expressed, using lots of punctuation marks and emphasis, my disbelief that he didn’t like every tiny thing about, apparently, anything. He says “here’s how these almost always go” and finishes by saying “Every. Single. Time.” Which shows that his arguments aren’t even internally consistent.

Trent:

I actually like KDE4  but I’m certainly not going to push it on someone who’s not interested.

There’s nothing wrong with liking KDE4. It’s not for me, that’s all. 

I’ve tried engaging with the developers at the Amarok forums, they are not interested in any critiques or comments on the UI. Politeness gets you ignored or patronised, frustration gets you deleted. Unless you’re part of the echo chamber don’t bother.

*nods*

I found that most of the issues I had with KDE4 and Amarok, when I looked around where one would submit such feedback, were things that were there by design, and would never be “fixed”, because they weren’t broken. Things like this convinced me it was best to just move on entirely. Despite what any of them said, they weren’t really interested in feedback that didn’t reinforce what they were doing.

They even come here and try to offer “help”. 

If they truly wanted to help, they’d have done what the little team working on Clementine has done. Port Amarok 1.4 to Qt4 to take advantage of some of Qt4′s features, and make it worthy of the name, rather than… whatever it is they’ve done with it.

That would have helped. Trolling blog posts about replacement applications doesn’t help at all. It just makes me mad. 

I actually don’t mind they have their vision and are somewhat single minded about it – that’s their prerogative as the developers, its this pretence that they are open to feedback that is quite irritating and the tendency to evangelise.

Yeah, that’s exactly what spins me up so much. Here’s how these almost always go.

1) Wait, you don’t LIKE *whatever*???? Inconceivable! Why?
2) Well what don’t you like about it?
3) Oh. Well, I think you don’t like it because youdon’t understand it.
4) Oh. In that case then, you don’t like it becauseyou’re resistant to change.
5) Well, I’m going to assume you’re stupid then, since you don’t like it. Here’s how to use it.
6) Why are you getting mad at me? I’M ONLY TRYING TO HELP.

Every. Single. Time.

Meh – Amarok is dead (for me anyway). Long live Clementine. And RhythmBox. And Banshee. And Bangarang … you get the idea 

Yeah. Amarok died with 1.4 for me. I’ve moved on, particularly now that it looks like a replacement (in Clementine) is in the works. It might take a while for it to catch up, but I’m pretty hopeful that those of us still using Amarok 1.4 can move on from it soon enough.

Thanks for the comment. 

Update 4: Valorie Zimmerman wrote a nice reply on Trent’s blog encouraging him to work with others, instead of simply spewing bile. Trent’s idea of responding to such a nice comment is to entirely miss her well-articulated point, insult me some more (to be expected), continue to insist that my comments were inflammatory and “intended to start an argument”, and then command her to “move on”:

Valorie:

You’ve completely misconstrued Jefferai’s response to you. Why the hatefullness?

Really, everyone here agrees on two things: we love music, and we love free and open software. So why not use the player you like, and contribute to that if you can. F/OSS works when people work on the projects they love.

F/OSS is just dragged down by hating on people and projects. Please stop doing that. I love Amarok, but point people to Clementine (or other players) if they want help finding alternatives. No project can meet all needs, or keep everyone happy. Enjoy diversity!

Trent:

I love Amarok, but point people to Clementine (or other players) if they want help finding alternatives. No project can meet all needs, or keep everyone happy. Enjoy diversity!

That was precisely the entire point of this article. To point people to Clementine, and to provide some reasons “why”. I think I did a reasonable job of that, unless you read a different article than I did.

My responses to Jeff were responses to off-topic, inflammatory comments intended to start an argument. I’ve already admitted that I should not have responded to him. It was my fault, I’ll freely admit that. Move on.

Recently, I have been contributing to another project outside of KDE: the Prosody XMPP server. Prosody is fairly new (the project began in 2008) and is written in Lua. I’ve been using it for a long time now, and it’s a really nice, rock-solid server. The code is also pretty hackable, being both modular and fairly clean.

My impetus for working on it was that at work we have been using the OpenFire XMPP server. It’s a big Java beast and is all but unmaintained at this point. Since I’ve had good experiences with Prosody and it already supported most of the functionality I needed, I decided to see how it handled a migration at work.

One thing it *didn’t* support was pluggable authentication. The basics for this were in existence, but the code was not yet right. After some pestering, Matthew Wild and Waqas Hussain, the two main Prosody developers, fleshed out the API necessary, and I began to work on separating out the built-in authentication code into a plugin that could be swapped out for other providers.

Once this was working, my next step was to attack something that has been bugging me on various XMPP servers to this point: plaintext password storage. To the best of my knowledge, every other XMPP server stores your credentials in plaintext or via easily reversible encryption with the key stored with your user account — in essence, simple obfuscation. The reason is because of the XMPP standards’ mandate that SASL PLAIN and DIGEST-MD5 be supported authentication methods. While SASL PLAIN transfers the password in cleartext (over the encrypted tunnel between client and server), thus lending itself well to comparison against a hash, DIGEST-MD5 does not. It consists of multiple possible authentication schemes based on the abilities of the client/server, and with different tranferred credentials depending on which scheme. It would require at least four saved tokens for every user, to be used depending on the mechanism the client and server negotiated, and updating these when passwords changed was not necessarily trivial.

So, I decided to make a tradeoff — you can store a plaintext password on the server and have DIGEST-MD5 available, or you can be limited to PLAIN and have a plaintext password go through the encrypted tunnel between client and server but have it hashed in the server’s storage. Personally, since I trust the server I’m connecting to, I’d rather have the latter. ejabberd makes a claim here that it’s more secure to have passwords sent in hashed form over the network and stored in plaintext on the server, but I think this claim is not well-reasoned. Like in many cases, security in this instance is relative. If you’re using SSL/TLS communications, which is by far the norm between XMPP clients and servers these days (in fact, in most clients you must explicitly enable plaintext authentication over unencrypted streams), the passwords are already encrypted when sent over the network — so unless you fear a MITM attack, you’re not going to be vulnerable to snooping. But you *will* be vulnerable to someone hacking onto your server and simply reading your plaintext passwords from a file, and that’s bad. I think that ejabberd’s claim is out-of-date now that encrypted communications between XMPP clients and servers are the norm, and you’ll be better off in most cases with having the one place the password is permanently stored be in a hashed form. (As you’ll soon see, you’re about to be able to have your cake and eat it too, anyways.)

The next step was deciding which hash algorithm to use. There’s something called SCRAM, which stands for Salted Challenge Response, and is an upcoming SASL authentication paradigm. It uses an open standard hash function called PBKDF2, which basically works like this: take the password, salt it with a large random series of bits, and then run it through the SHA-1 function. 4096 times. Rainbow table *that*. If you like, you can swap out which hash function you use, so you could use SHA-256 or some other function, and you can do it more than 4096 times if you like, too.

By limiting what information is passed between server and client, using secure hashes like PBKDF2, and never transmitting the password in cleartext to the server, SCRAM provides a SASL mechanism that is both highly secure and more standardized than DIGEST-MD5.

Now, back to the hashed password backend. Support for SCRAM is beginning to appear in some XMPP clients, and support was mostly coded into Prosody as well, so I asked the guy who wrote it, the excellent Tobias Markmann, to expose some bits of the SCRAM capabilities such that I could use the PBKDF2 functionality to store password hashes, as well as allowing arbitrary salts and iteration counts to be defined. He did this, and I coded up the backend, and suddenly Prosody became the first XMPP server, to my knowledge, that can securely store your password server-side.

The backend provides seamless upgrades from the plaintext backend; it will simply take the password that currently exists for the user, generate a random salt, and create the hashed credentials, removing the plaintext password.

Tobias is currently taking it a step further: he’s modifying the backend to store bits of information required by the SCRAM algorithm instead of the direct password hash. What this means is that if you end up using a client that only supports PLAIN authentication, those bits can be generated by your password on the fly and used to verify against. However, if your client supports SCRAM authentication, the backend will now be able to serve up the bits necessary for SCRAM-SASL. So you’ll have secure server storage *and* a secure exchange from client to server (in addition to the normal SSL/TLS tunnel). Tobias is, I believe, working as a SoC student on Psi — so I’m sure it’ll be working between Psi and Prosody pretty soon, if he doesn’t have it done already.

The hashed password backend isn’t currently the default, but if you’re running Prosody you can enable it by adding auth_internal_hashed to your modules and defining (in your host) the line

authentication = "internal_hashed";

Once it’s tested more thoroughly, and especially when SCRAM support is finished and working in more clients, expect it to become the default backend.

First off, I need to give a big shout-out to Nokia. A while back, they donated ten n900 devices to KDE developers who submitted proposals describing what they would work on. I was one of the lucky winners, and this provided both the motivation and means to start working on an Amarok Mobile.

I actually was not initially supposed to use this for Amarok Mobile development — I had sumitted a proposal to bring functionality similar to the beloved* Popup Dropper into a library for Maemo programs. Problem: having never used Maemo before, I didn’t realize it had longpress functionality (my previous phone was an iPhone).

So, not wanting to reinvent the wheel needlessely, I instead decided to start working on exactly what everyone had assumed/thought/wanted me to be working on when they heard I was one of the n900 recipients: Amarok Mobile.

Below is a record of some of the work that has been done, and that remains to be done.

Core work

In large part due to its very rapid pace of development, Amarok’s code base has had a habit of growing haphazardly, and one of the results has been a lot of incestuous code…things coupled when they really shouldn’t be, code being referenced and implemented all over the place, and so on.

As a result, one of the first things I did was to work on separating out code that provide core functionality — things like podcasts, playlists, our metadata system, collections, and more — and boil them down to the basics. The rule for the core library is: no GUI code, and nothing inside core may reference any files outside of core. The point is that core should comprise the essence of Amarok, and much of what’s left are the possibly platform-specific implementations. In addition, I standardized namespaces for the parts of core, standardized include headers (which makes it much easier to handle file moves later, since they are fully-pathed from the top level source directory), and had absolutely countless compilation cycles. It was an absolute fsckload of work.

However, I think it paid off. We now have, inside src/, a core/ directory and a core-impl/ directory. If we studiously keep core/ separated, it will provide an excellent base on which alternative GUIs and implementations of those core classes can be built. In addition, more things will be moved there, or at least the paradigm will continue being followed; for instance, our great contributor Erik Hovland has been making progress on separating out the GUI layer from our Services framework so that we can have the great services like Ampache and Magnatune and Last.fm and so on be relatively easily activated on other platforms.

SQL

Amarok 2 currently uses MySQL exclusively, allowing you to either use a MySQL server or libmysqld for an embedded database. This has pros and cons, but the really relevant pro is the huge speed gains we get over SQLite. (Before people comment about how we must be doing something wrong: we’re not the only project that has seen serious speed problems with SQLite — I know that Quassel has had similar issues; and if you’re convinced that it’s our schema or some other such thing, you’re welcome to help us improve it. Many others have made this claim, and none of them have ever taken our invitation to actually help us improve things.)

So the obvious question comes up: what about Amarok Mobile? I’m not sure how realistic it is to expect that we can get libmysqld running efficiently on the n900, especially given its memory limitations. At the same time, SQLite is readily available. But, we want to avoid writing tons of custom code.

As it turns out Quassel has a really great storage abstraction using QtSql, and they use both SQLite and PostgreSQL. So, if we copied their really nice approach, we could maybe support SQLite with Amarok Mobile while reducing developer effort to a minimum (since none of us are SQL experts).

One of the big problems we’ve had with QtSql before is that the QMYSQL plugin can support libmysqld (except that it can’t; more on that in a second), but in one of the most boneheaded software design choices I’ve ever seen in my life, MySQL determines whether you’re connecting to a server or an embedded server based on which library your program *links* to. This is not a runtime option. Wut.

So — the obvious option is to bring to Qt what we do in Amarok — instead of having a QMYSQL driver that has to link to one of the other, have a QMYSQL driver linking to the server libraries and a QMYSQLE driver linking to the embedded libraries, and let application developers choose one or both. I began working on this, and have been successful in building both drivers, and have high hopes I can get it included into Qt 4.8 (there’s necessarily a lot of code duplication, so I’ll have to see what they say about that…I may be able to abstract some of it into shared functions, but anywhere that mysql in any case was used, I had to replace with mysqle).

Now, I said that it can’t actually support libmysqld. The reason is that you pass in options for the embedded server upon library init — but QtSql *always* initializes the library with null options. Which means, among other things, that it will use /var/lib/mysql (or whatever your compiled-in default is) for its database directory, which normally means that if you’re not the root user, you can’t actually do anything because it can’t create its files. Whoops.

I can’t break ABI or API, so I’ve been wondering what to do about this — some of the Qt guys suggested I create a new SQL framework with these drivers in it, but I don’t think that’s a great option. Instead, I plan on creating a new connection option (see the table here). Normally these are basically mapped to MySQL options and used like this:

db.setConnectOptions("CLIENT_SSL=1;CLIENT_IGNORE_SPACE=1");

This connection option will not be mapped directly to a MySQL option, but will instead take a delimited list of options, something like this:

db.setConnectOptions("CLIENT_SSL=1;CLIENT_IGNORE_SPACE=1;MYSQLD_INIT_OPTIONS=myoption1=myvalue1;;myoption2=myvalue2;;myoption3=myvalue3");

This would let you pass arbitrary options, and this list would be split and each of the options would be passed into library initialization. It’s API/ABI compatible and relatively straightforward, so it’s the best thing I can think of right now.

Future work

The next steps are to finish the QtSql modifications and get those submitted upstream. Maemo right now has Qt 4.6, but *eventually* will have 4.8…so when 4.8 is coming around, if my QtSql changes are in there, Amarok can try migrating to QtSql; to finish services GUI separation and separation of other parts of Amarok (like AmarokURLs), and to begin writing a GUI, ideally using the Qt Kinetic declarative UI library — when you’re starting from scratch, you might as well do it right. Fortunately some of these tasks will be much easier now that Qt is fully supported/included in the proper locations with the n900 PR 1.2 firmware release.

My time has become incredibly limited in the past month, so if you have time and are interested in helping, join us!

* Okay, I know one person that really, really hates it. And a lot of people that never really realized it was there. But a lot like it!

Previously, our advice was “touch the folder” or in the worst case “just do a full rescan.” Now there’s slightly better advice we can give. If you right-click on a folder in the collection setup dialog, and if the folder was already in your collection (i.e. you didn’t just check it a moment ago), it will give you the option to rescan that folder.

The text above that field has also been modified slightly to indicate that this is possible. Check it out:

Rescanning a Single Folder

This is in current git master and will be in the 2.2.3 release!

This number was at least three. At the very minimum, assuming the artist, album, genre, composer, and year had been cached, you had a lookup for the uniqueid, a lookup to check for whether the track was part of a compilation, and an insert. This number could be significantly higher in some situations; but these database queries were a large part of the reason that a scan might run, finish disk I/O, and then spend quite a long time eating up CPU time before it was done.

I looked at increasing the number of values I was pulling in using UNION queries, which was currently up to five values (depending on whether they were already cached). I also looked at prepared statements, which it turns out were not likely to give us a large speed bump and are an absolute and utter pain in the ass when using the C API as we are. (We use the C API for historical reasons; also, the official C++ API is not easily available in many distributions, and there are many unofficial contenders.)

Finally, I looked at ripping the guts out of the scan result processor and doing what  several of us devs had decided would probably be the best way to improve speed, but was at the same time the most challenging: replacing the SQL statements with local memory storage, populating these at the beginning of a scan and re-populating the database at the end. It wasn’t easy. I selected data structures very carefully for speed, looking at how often each query would be run and the time complexities of insertion and lookup into various types of structures.  In addition, because of a few complex queries using joins, the behavior we needed could only be emulated by using very complex data structures (one of them is a QHash<QPair<int, QString>, QStringList*>; another is a QHash<QString, QLinkedList<QStringList*> *>). I also wanted to minimize memory usage…I did some rough back-of-the-hand calculations which indicated that using my design, for the vast, vast majority of collections, memory usage during the scan was likely to go up by only a couple of megabytes at the very most, which would be reclaimed at the end.

In addition, I implemented logic to drastically decrease the total queries needed for insertion. Since I’m constructing the final queries from the data structures, I could easily do insertion of multiple values at once. So now the database is queried at the start to see the maximum query size it will accept; then values for insertion are appended to the query up until this value is hit, at which point the query is run and a new query begun. By default I think this is one megabyte for most installations, which means that instead of possibly thousands upon thousands of insertion queries (depending on the size of your collection), you’d have to have an absolutely extraordinarily large collection to have more than 25 or so queries for the entire scan. Since there is not only round-trip delay sending data to and receiving responses from the database but also the database must parse each query each time it’s run (the purpose of prepared statements is to reduce this), then this makes a drastic difference. To put it another way, the number of queries per track has gone from a minimum of three to an asymptotic value of zero.

Overall, the work seems to have paid off rather nicely. Benchmark reports that I got back indicated anywhere from 30% to 300% gains in the total time for a scan, depending on size of collection and your particular I/O throughput/CPU/etc.

I call this the Ultimate Speed Bump in the title because there is far less that can be done from this point on to increase speed, at least as far as I can currently tell — this was the Big Mama, the elephant in the room. But hopefully it will increase scan processing enough that further increases won’t really be so much of a necessity anymore.

One other data point: the database is now case-sensitive, which is something we’ve intended to do for a long time but is now being done. It’s been a very long-standing feature request that we’ve finally implemented (we wanted it too), and thankfully it wasn’t much work to do this and change the appropriate points in the rest of the code.

(By the by, this will all be in 2.2.1 — enjoy!)

Professor Bourne is well-known in his field for contributions to open-source bioinformatics software and is a leading advocate of open access to data. Quoting from the UCSD News Center:

Bourne is co-founder of SciVee, the Web 2.0 resource dedicated to the dissemination of scientific research and science-specific research networking. Launched in late 2007 as a collaboration between the National Science Foundation and the San Diego Supercomputer Center (SDSC) at UC San Diego, SciVee has been used by hundreds of thousands of students and professional scientists as a means of learning and sharing their research through online science videos that supplement peer-reviewed journal articles, stimulate discussion, and promote collaboration. SciVee earlier this month announced a number of significant upgrades to its site, along with the addition of 32 new science categories.

Open data access and open source share similar goals, and Professor Bourne’s discussion of his experiences with open data access should prove informative and interesting to all.

First up, and the lesser of the two: If a directory is encountered multiple times during scan, the scanner will now only scan them once. There are a few corner cases where this could happen (for instance, if you have a top-level directory specified as well as one of its subdirectories, and the subdirectory’s name changes, changing both its and its parent’s mtime). Not too useful for most cases, but was implemented while working on…

Second, and the biggie: When doing an incremental recursive scan, Amarok will now no longer scan subdirectories whose mtimes have not changed. This is big news if you have a large, flat collection (which I dislike myself but some really enjoy…to each their own). For instance, in a setup like 3,000 files in your main folder, plus a few thousand files each in a few subfolders for a total of 10,000 tracks, if you added a single file to the top-level folder and had incremental recursive scanning turned on (the default), you’d cause a rescan of your entire music collection — all 10,000 tracks or so. Now, if you add a single file to the top-level folder, you’ll only cause a rescan of 3,000 tracks…which is still a lot, but even more of a reason to use a hierarchy.

This should help scanning time even for those with hierarchies, as adding a new album to an artist won’t cause the artist’s other album folders to be rescanned, or adding a new artist to a genre won’t cause all the artists and albums in that genre to rescan.

It’s really the proper way to do things, but it required some changes to the data sent between Amarok proper and the collection scanner, so had to be done carefully (as far as I am currently aware I didnt’ cause any regressions). Regardless, I’m glad to have it in there and working, and hopefully you will be too.

I’d heard these whispers recently about “whenever I add or remove an album from the collection and do an incremental update, my collection gets messed up.” Okay — we’ve heard these whispers for a long time, but there were so many other things that had to be fixed first that it wasn’t clear whether this was a symptom of a different problem or a problem of its own…and it could always be fixed with a full rescan.

However, with the collection being much more solid these days and with this being the only super-visible bug left that I knew of, I decided to tackle this. It turns out that, once the rest of the collection was behaving, this wasn’t that hard to find…it just took a lot of debug tracing, because it wasn’t obvious. Since the cause was the wrong field being used in a DB query to remove some tracks during an incremental scan, with anything other than a very tiny test collection it could quickly start to make no logical sense what was happening.

The reason I say the bug is historical is that the code causing it has been there since May 2008. While admitting that provides some fodder for naysayers and haters to harp on Amarok for such visible and longstanding bugs, I prefer to take the other approach: it means that all the various issues users have found since the total rewrite that was 2.0 *are* being found and *are* being solved. Some of them just take some time; it’s a *lot* of code. But the proof is in the 2.2 ChangeLog.

I managed to sneak in the fix a day before 2.2 was tagged, so when Amarok 2.2 comes out scanning (both full and incremental) should be really quite solid. In fact, since this has been fixed in git, I’ve yet to hear of any more problems, just lots of happy users. It’s just another way that the (very close!) 2.2 is going to *rock*.

Then, a little while later, I started getting bug reports of peoples’ music disappearing from their collection, and requested some of the reporters send me some files. One of the users did so, and I found something curious in his tags (if I had a penny for every time I’ve personally seen users have something odd or strange in their tags, I’d have…well, a few dollars at least). Several of his files had full MusicBrainz tags — with absolutely no data populating them, meaning that the MusicBrainz identifier (and all other MB data) for all of those files was ending up the same (blank) and Amarok was thinking them the same file.

It was a quick fix (use generated non-embedded AFT IDs when the MB tags are empty) but just adds to the evidence that you can never, ever trust users’ tags. Also, that users that use your Git-based version or betas really rock for finding this stuff before release…so in case I don’t say this enough: thanks users!

Full details are here.

As it turned out, there was a "Kde" network — but something was odd. To join a work network you have to have an email address affiliated with the network. KDE owns kde.com and kde.org — so who was this? The only other "KDE" I could find that seemed like it would be legit was the Kentucky Department of Education, and I rather doubted it was them, because they would likely have used all-uppercase KDE as well. So I started an inquiry with Facebook, trying to figure out if either it was someone squatting on our name (and trademark) or whether it was some legit organization — in which case, would they mind donating the network to us?

After several months of back-and-forth with the people at Facebook, who were very nice (if a bit slow ), I’m happy to say that we’ve regained the KDE network (properly capitalized) as our own. I still don’t know the whole story as to who was there before, and never will due to their privacy policies, but I’ll say this:

• If you were in the "Kde" network before and Facebook asked if you would mind donating it to us, and you did, thanks so much!
• If someone was simply squatting in the "Kde" network before, then thanks, Facebook, for kicking them out!

To join the network, go to Settings -> Networks, and enter KDE and your kde.org email address in the appropriate fields.

First, the schema/table changes.

1. We’ve seen some issues where people have, for whatever reason, ended up with InnoDB tables instead of MyISAM tables. This is probably the result of their DB being created long ago before we were explicitly telling the mysqle startup to skip InnoDB. This mainly causes a problem because some columns cannot be as wide as we’d like them to be when using InnoDB. So, the first thing being done is that an ALTER TABLE is being forced on every table to explicitly convert to MyISAM. In addition, ENGINE parameters are now used during table creation to be more explicit in the future.
2. Some of you might have seen complaints in the debug output about indexes not being able to be created due to a max key length, which by default in MySQL is 1000 (compile-time option). So, some columns have had their widths adjusted so that all indexes are now successfully created.

Now, the other changes:

As we added more features, scanning got slow. Like, really slow. You’d spend more time running SQL queries than actually scanning your files. So I’ve been aiming to change that.

Over the past week I’ve committed changes that remove, per track, anywhere from 1 to 6 SQL queries. The exact amount is highly dependent on your file set, but there is a minimum of one less SQL query per track. If you’ve done a lot of file moves and AFT kicks in, it’ll be an even more massive speedup. I’m going to try to do some further tuning, but already results are looking positive.

Nikolaj has reported that his scan time went from 68 seconds to 18 seconds — more than 3x faster. Mikko didn’t notice a speedup, but he said that whereas scanning used to peg his CPU at 100%, it no longer does so. What I want to know is: how does this affect *you*?

If you want to help, do the following:

1. Backup your DB. If you’re using external MySQL do a mysqldump, if you’re using internal MySQLe backup the mysqle folder in the Amarok data directory.
2. Update to a revision from a week ago…say, 995000.
3. Wipe your DB.
4. Start Amarok — it will do a full scan because of the empty DB. Time it as it does the scan.
5. Repeat steps 3 & 4, so that you can see what the time is like after caching.
6. Update to current trunk (at least 998470).
7. Repeat step 3.
8. Repeat steps 4 and 5.

Then leave a reply here with your values. If you watch your CPU during each of the scans, report that here too. Thanks!

Please note that this is *not* a post about why KDE is migrating to Git, why this is a good idea/bad idea/neutral idea, etc. This is purely discussing the hosting aspect of Git.

First, I would encourage you to read this kde-scm-interest mail, which I sent to the list on July 2nd. It goes into a good amount of depth as to why Gitorious.org could be beneficial for us, and the rest of this post will assume that you have read that email and the others in the thread, as it will simply update the information therein.

On Monday a large group of interested people, including KDE sysadmins and the guys from Shortcut AS, went to lunch to discuss the technical issues. The output from that discussion is as follows:

• The vast majority of those present feel that Gitorious.org would be the best choice, with the following being the main reasons:
• Shorcut could provide a SLA (Service Level Agreement) guaranteeing a minimum level of service, such as uptime and available bandwidth, providing professional hosting services and easing burden on our system administrators.
• As David Faure noted, user account creation is becoming a large burden on our system administrators, which is not something that we would have to administer if using Gitorious.org.
• It should be noted that the above was not a unanimous opinion.
• KDE does have infrastructure and bandwidth; it could keep one read-only Subversion server available for historical reasons, and convert the rest to serve as backups or possibly load-balancers. Or to put it in a more general fashion, KDE can reduce hosting costs (which will likely be covered by sponsors) by working with Shortcut. It is not a question that this could be done, but rather what the right method would be for doing so.
• The Gitorious developers have a feature branch where they have already fixed one or both of the current showstopping bugs relating to rights within the shared Git repository. They have said that this should be merged into mainline within a week (not sure if they meant a week from then, or from the end of GCDS).
• The hosting could be set up in such a way that it can be accessed via git.kde.org.
• Post-commit-hook functionality will be available; the Shortcut guys are currently working with us to determine how we can migrate or emulate pre-commit-hook functionality.

We have two projects that are chomping at the bit to get onto Git ASAP: Amarok and TagLib. Amarok will be converted first and will serve as the initial guinea pigs to iron out any issues. Barring any major issues being found, TagLib will be converted in short order.

I hope this gives everyone a better idea of KDE’s Git-hosting plans. If you haven’t checked out Gitorious.org, I encourage you to do so; it’s made huge leaps and bounds in the past six months and has become quite a great tool.

Please direct any questions or feedback to the kde-scm-interest mailing list at: kde-scm-interest at kay dee eee dot ooo arr gee, not to the comments section on this blog.

In revision 992942, I finally fixed a bug that has kept AFT working for the playlist in certain situations (although it had previously been working for both saved user playlists and statistics). This means that if you have a track in the playlist, move it to another location, and it is then scanned in that new location (remember, kids, it uses folder mtime to determine whether to scan a folder, so when in doubt do a "touch ."), the track in the playlist should remain valid and play the song in the new location. As the playlist use case was one of the initial reasons for the development of AFT back in Amarok 1.4, you can imagine I’m happy that it’s finally (seeming to be) working again in all scenarios, instead of failing in certain situations.

Well…you were wrong.

It may be a bit later than planned — we wanted to have it in time for 2.1, but it didn’t happen — but as of revision 984572, there is now support for storing an Amarok database on a MySQL server instead of the embedded MySQL database.  There’s no configuration dialog in the GUI yet, but it’s pretty simple to set up, as explained below.  All you have to do is add a few things into your amarokrc file and make a valid user on the MySQL server instance of your choice — you don’t even need to create the database yourself.  (In fact, you shouldn’t — you should let Amarok create the database so we can ensure that the character set and collation are set right.)

Here’s how to do it.

1. Update to at least r984572 (of course, updating to the latest revision is probably your best bet).
2. Wipe your build dir clean and rebuild.  Not necessarily necessary, but as 47 files were changed in that commit, it’s not a bad idea.
3. After install, run kbuildsycoca4 –noincremental, just in case. 
4. On your MySQL server, run a command like: "GRANT ALL ON amarokdb.* TO ‘amarokuser’@'localhost’ IDENTIFIED BY ‘mypassword’; FLUSH PRIVILEGES;"  Be sure to substitute for "amarokdb", "amarokuser", "localhost", and "mypassword" as appropriate.
5. Open up your amarokrc file, usually in ~/.kde4/share/config/amarokrc.  Add a [MySQL] section:
   [MySQL]
   UseServer=true
   Database=amarokdb
   Host=localhost
   Password=mypassword
   User=amarokuser
6. Close the file and start Amarok.  It should create the database and start a scan of your files.  If you want to switch back to the embedded collection, simply set "UseServer" to false.

Pretty easy!  Be sure to let me know if you have problems — file a bug and assign it to "mitchell" at a domain of "kde" plus a dot plus "org".

We’ve had various bug reports over the years relating to character sets and collation, causing issues with matching searches for music or mis-sorted items.  Well, hopefully no longer.

When you update to 2.2 (recent SVN users, see the note at the end of this post), your Amarok database and tables will be converted to use the ‘utf8′ character set and ‘utf8_unicode_ci’ collation as default for any table or column created from this point on.  Every single text/varchar field will also be converted through a two-step process to use ‘utf8′ as the character set (the data inside was always UTF-8, but there was a possible mismatch between what the data was and what the database thought it was, if your mysql wasn’t built to use ‘utf8′ by default).  In addition, the character set used when talking to the embedded server (the protocol in the socket) will be ‘utf8′.

Fixing this mismatch between what the server might have been using for character set/collation and the data we’re putting in there should hopefully ensure that sorting and tags work very well for our users with some files wth non-Latin1 tags (probably just about everybody these days).

* Recent SVN users: if your build date is earlier than this post I’d recommend wiping your mysqle directory (not just a full rescan), as the initial commit of the updating code contained a bug that could possibly cause trouble down the line with user playlists…but you bleeding edge users should be expecting database wipes every now and then 

The good news: in current SVN (and thus 2.2) the amarok_afttagger executable will also now handle FLAC and various Ogg-contained formats. Another piece of good news – the amarok_afttagger executable is now contained in the amarok-utilities package, and thus can be run on headless machines without X!  And lastly — AFT now works with user playlists, so you can move your files around (keep in mind the caveats if you’re not using embedded AFT tags) and your playlists will always stay current, in addition to statistics and The Playlist.

The bad news? Something is currently a bit broken somewhere deep inside with Observers which means that The Playlist will only update with the correct new URL once (the metadata observers seem to die after that).  This doesn’t seem to be AFT specific but rather seems like a bug that AFT is exposing.  Closing Amarok and reopening it will cause the proper new URLs to be in the playlist.  I’m working on trying to fix that.

(Important note: it writes into the FLAC Xiph comment.  This is the only metadata type actually required by the FLAC spec, and thus is the proper place to put it, but a lot of FLAC files incorrectly only have ID3v2 tags, so depending on the tagger you’re using you may only see one or the other.)

Anyone interested in scanning their collection locally instead of across a network connection, or keeping their collection up-to-date when Amarok is closed, should definitely give it a read!

http://amarok.kde.org/wiki/Batch_Mode

This means we have a lot of great work ahead of us.  The first order of business is that everyone that wants to be a mentor needs to sign up at http://socghop.appspot.com.  You’ll need a Google account (which does not have to be a Gmail account).  Click the link that says "Apply to become a Mentor" and proceed as instructed.  You should also ensure that you’re on the kde-soc-mentor mailing list.

Students: now that we’re accepted, get to it with contacting mentors!  The application period opens March 23rd and closes April 3rd, so it’s not a huge amount of time to get all the details of your proposals worked out.  Be sure to check the ideas page at http://techbase.kde.org/index.php?title=Projects/Summer_of_Code/2009/Ideas, or contact the appropriate mentor if you have a different idea for a project that you’d like to work on.

I’ll be sure to keep everyone informed of events as they occur and deadlines as they arrive.  Here’s to a successful summer!

]]> http://jefferai.org/2009/03/gsoc-2009-were-in/feed/ 0 http://jefferai.org/2009/03/gsoc-2009-application-submitted/ http://jefferai.org/2009/03/gsoc-2009-application-submitted/#comments Tue, 10 Mar 2009 21:49:00 +0000 jefferai http://jefferai.org/?p=1050 Today the application for KDE to participate in GSoC 2009 was submitted.  They check out our ideas page when they’re evaluating applications, so be sure to get your ideas up there!  We’ll hear back by March 18th.  Some other upcoming dates:

• March 18-23: Students discuss applications with mentoring organizations (you can and should do this earlier, of course)
• March 23: Student application period opens
• April 3: Student application period closes

Of course, this hinges on us being accepted…but past years have shown that we have a good shot.

]]> http://jefferai.org/2009/03/gsoc-2009-application-submitted/feed/ 0 http://jefferai.org/2009/03/licensing-to-kill/ http://jefferai.org/2009/03/licensing-to-kill/#comments Wed, 04 Mar 2009 06:43:00 +0000 jefferai http://jefferai.org/?p=1049 Could FLEXlm be one of the world’s worst-designed programs?

They’ve just rechristened it FLEXnet Publisher, and I can only think it’s to try to get away from existing FLEXlm stigmas.  It’s so bad that according to a VMware engineer I spoke to on the phone a few months ago, in the next release of ESX (whatever the new name is going to be) they’re ditching it to go back to their own serial-number based scheme, entirely because of a large amount of hugely negative customer feedback.  This is only one major release after they switched to it.

Let me describe the structure of it (if I were to go into the user interface issues, this would become far too long a post, so maybe another time).  There’s a management daemon, called lmgrd, and vendor daemons.  These vendor daemons are (often along with the management daemon) provided to you, often without an installer and simply as a bunch of files.  There’s also quite often a node-locking generator (based on things like hostname, MAC address, and all manner of things easily faked in a VM) provided to you by the vendors, which as far as I can tell is different for every vendor.  This is possibly so that they can build in their own criteria, but more likely because the FLEXlm people are lazy and can’t be bothered to provide a comprehensive set of criteria on their own.

Now, why is there a vendor daemon and a management daemon?  Good question, and I don’t know the authoritative answer, but from using it I can make an educated guess or two: either to allow vendors to validate some part of the licenses in their own manner, or to simply confuse and annoy the hell out of you.  See, the different vendor daemons are built against some version of the FLEXlm software, and they may, or may not, have been built against the same version.  This may, or may not, cause problems, including crashes.  It may explain why every time the license server reboots (thanks, generally, to automatic Windows updates), lmgrd fails to start up successfully.

Now, you can run multiple instances of lmgrd — if you can figure out how to get it installed so that it starts up on bootup, since there isn’t much in the way of help in the official help PDF, and only one of the three vendors whose licenses I deal with actually provided an installer (thanks, VMware, I’ll be using your installer long after you’ve switched off of FLEXlm).  This lets you have each license under a different copy of lmgrd, perhaps so that you can attempt matching versions, but mainly so that if one of the lmgrd instances crashes, it will only take down that license.  (Actually, it’s because in this configuration you need to run them on multiple machines, so if one machine goes down, most of your licenses stay up.  But I know what they really meant.)

Alternately, you can combine licenses into a single file, a process that the manual warns is time-consuming and error-prone.

Finally, you can simply have all your licenses be handled by a single lmgrd instance.  Now, this isn’t really a bad idea, except for port numbers.

See, each vendor daemon needs to run on its own TCP port number.  You can specify the port number in the license file, except when you can’t because some vendor hardcoded it into their software.  If you don’t specify a port number in the license file, the vendor daemon will be given a port number starting at 27000 on up, giving you an address like 270...@mylicenses.thissucks.com.  Now, because everyone likes autoconfiguration, if in your program you leave out the port number (giving you something like @mylicenses.thissucks.com), then it will automatically search ports 27000 through 27009 for a matching vendor daemon.  Unfortunately, I haven’t seen a single vendor product that doesn’t puke on such an address, claiming it is invalid because of faulty validation, even when the FLEXlm manual in front of me says it’s perfectly legal.

If you don’t then modify your license files to specify ports (which you have to remember to do every time you get a new license file), then you better be aware of when the license server reboots, because the ports are given out in the order that the services come up.  So if you’re in my situation, where the first of the vendor daemons always crashes the first time it tries to load on bootup, then the other two will have their normal ports decreased by 1.

I haven’t even gotten to things like functions to reread license files that don’t actually reread them, a UI that doesn’t tell you if a server is running or stopped until you hit the buttons to try to run it or stop it, a status window that displays a large amount of license data in a tiny, non-resizable window, and more.  The product feels like it was developed in 1988 (it was) and is a study in market leader stagnation.  I would be very surprised if they have a single developer left and haven’t fired everyone to just sit there and watch the money roll in.

If license servers are necessary for some product you’re creating, there are other license servers out there that simply run on a well-known port and don’t require any of this idiocy.  Unfortunately, in my experience I’ve seen more companies switch to FLEXlm than away from it. I hope this isn’t an industry-wide trend.

Where’s the KDE angle?  There isn’t one specifically, other than this: if someone in the KDE community thinks this is a good design, or can see themselves designing such a scheme, please just leave.  Now.  (Don’t worry, I’m sure this doesn’t apply to any of you. )

]]> http://jefferai.org/2009/03/licensing-to-kill/feed/ 11 http://jefferai.org/2009/02/amarok-power-user-feature-batch-mode-collection-scanning/ http://jefferai.org/2009/02/amarok-power-user-feature-batch-mode-collection-scanning/#comments Fri, 27 Feb 2009 23:58:00 +0000 jefferai http://jefferai.org/?p=1048 A long-requested feature has been a way to decouple Amarok’s collection scanning from its GUI.  There are various use-cases for this.  For one, it can actually help us with debugging, by allowing us to control the inputs into the scan parser.  For another, many people have all of their music stored on a single machine, and would like to do the scanning locally where it’s fast instead of on their e.g. laptop running Amarok, where it’s over wireless and slow.

Yesterday and today (as of r933010) I put half of the solution into trunk.  I say half, because full collection rescans are now supported in batch modes, but I am still working on the methodology for incremental scans (I have a few ideas, but have to sort out which is the most reasonable/doable/makes the most sense). Below, I’ll explain how to do it.  Keep in mind that this is designed to be (lightly) scripted, not done by hand…so it can be done by hand (which I did during testing) but it has some safeguards in place so that if you script it, and forget about it, Amarok still works normally.  However, there are actually some interesting things you can do now if you script the scanner…

One important thing to note: the scanner requires various bits of Amarok code, mainly centering around the extra taglib plugins we support.  So you’re unlikely to have it work on a machine that doesn’t have A2 installed, and certainly won’t be able to compile it without the rest of the Amarok source, although you may be able to get it to work in a binary-only fashion with just kdelibs and taglib…YMMV.

So, here’s the flow:

1. Run amarokcollectionscanner with the -b or –batch option.  Although this doesn’t directly modify the output, it sets some internal flags (some for safety, some to enable other options).  The output goes to stdout; save this output as amarokcollectionscanner_batchfullscan.xml (yes, it must be that specific filename).  You’ll probably want to use the -r flag too…check –help for more options.
2. Cool feature: if your scan is of files that have a different mount point on the local machine than the machine that will be using the output, add the –rpath option and give it the mountpoint of the directory on the consuming machine.  For instance, if on the local machine your music is at /opt/music, and this directory (music) is mounted as /mnt/music on the remote machine, run amarokcollectionscanner from /opt and use –batch –rpath /mnt/music.
3. Cool feature: You can scan multiple directories and simply append the output to the end of the file from step #1.  Yes, directly append, including the xml version/encoding header.  Amarok’s scan manager will detect and handle this.  You can even scan directories that are not defined as collection directories in Amarok (these entries will work fine, but the next time you do a full rescan from within Amarok without using the saved output, they will be gone).
4. Place the amarokcollectionscanner_batchfullscan.xml file in your Amarok data dir.  Usually, this is ~/.kde4/share/apps/amarok.
5. Trigger a full rescan from inside Amarok (in Settings->Collection).  It should be super-fast.    Note that this step deletes the amarokcollectionscanner_batchfullscan.xml file.  This is on purpose, so if you want to keep re-using it, you’ll have to keep copying it over.

As you can see, it’s designed for scripting, but it’s not terribly complicated even to do by hand.  I’ll post more as I get incremental scanning working, and will at some point put a Wiki page up that has all of these instructions in a less transient form.

]]> http://jefferai.org/2009/02/amarok-power-user-feature-batch-mode-collection-scanning/feed/ 3 http://jefferai.org/2009/02/camp-kde-videos-come-and-get-em/ http://jefferai.org/2009/02/camp-kde-videos-come-and-get-em/#comments Wed, 25 Feb 2009 18:41:00 +0000 jefferai http://jefferai.org/?p=1047 After a month’s delay, they’re done.

In the interim (after the delays mentioned in my last post on this topic), my poor, underpowered desktop machine has endured transcode after transcode (from the original source material) and my Internet connection upload after upload as I tried to figure out just why Blip.tv wouldn’t work with X or Y.  (In fact, I can quantify these: X is Vorbis/Theora, which produced awful audio and very desynchronized audio/video upon their conversion to .flv; Y is a lot of things related to the original anamorphic encoding of the videos, which Blip.tv can’t handle, and finding the right combination of settings and flags and adjustements to make the aspect ratios come out so that everyone didn’t look like Gumby®.)

This was followed by a few days of uploading; Blip seems to max out uploading speeds somewhere between 100kbit and 200kbit, so uploading almost 6GB of data, one chunk at a time, took a bit.

The end result is thirteen videos, each Xvid-encoded (at least it’s OSS, although patent-encumbered…see Vorbis/Theora problems above) with a Matroska container.  You can get to them through my show or to individual videos directly (these might not be in strict as-presented order):

• Welcome!
• Diversity in KDE
• Akonadi
• KDE and Business Software
• KDE4 and MS Windows
• libplasma in Applications
• Accelerating Graphics
• CMake / CTest / CPack
• Taming the Leopard
• Bringing the Free Desktop to the Mobile World
• KDE & Distros
• KDE-Games, KDE-Edu, and Avogadro
• A Case for Open-Source Coders in the Enterprise

Enjoy!

The title says it all.  They’re available from my PPA.  Enjoy!

The Bad News

I’ve run into a series of setbacks regarding getting them prepared for posting.  The first was that the Monday I got back from Jamaica, I had the following schedule at work:

• 9:15 AM: Check email
• 9:16 AM: "My details have been confirmed for my upcoming trip?  What trip?"
• 2:00 PM: At the airport awaiting departure, thinking that I *knew* I should have checked work email sometime over the weekend…

Regardless, I did manage to have some of the files with me, and did actually get a few of the videos done during that week that I was away.  I then got held back a bit waiting for the videos on Wade’s camera to arrive — they were 16GB or so in total, and so he had to fetch them off the tapes, encode them, and get them uploaded to me.  After that, I started to put Wade’s videos together with slides, only to find out that the videos would not properly work inside kdenlive.  Or so I thought.  After a period of long transcodings that I performed during my free time over the last few days, I find out this morning that the issue with the videos was a massive dose of PEBKAC.

There’s only one other piece of bad news, which is that due to a bug in (ffmpeg? libmlt?) kdenlive’s Ogg/Theora export is borked.  So, I’ve ended up coding in XviD, which while using a patented algorithm is at least otherwise free software.  I’ve thought about transcoding everything to Ogg/Theora afterwards, and I may end up doing so before posting them up, if the quality is okay.  And before you ask — it didn’t seem right to post them up without having Wade’s keynote up there to kick things off, which is why I’ve not posted some of the ones that are already done up before now.

The Good News

The good news is that 11/13 of the videos are done, and I forsee no obstacles in getting the rest of them to behave.  So I am hopeful that I can start getting them up to Blip.tv within a couple days.  Thanks for bearing with me.

Anyways, one of the reasons I’m using it is to splice the slides into the videos, because they’re just not readable inside the videos for the most part. So I needed a way to turn each slide into some sort of an image.

It turns out that OO.o doesn’t have this capability natively, but some users on the OO.o Forum came up with a script at the bottom of this page to export each slide into PDF. I modified it to do the following:

• Export to JPG instead of PDF (PNG export didn’t work)
• Add extra 0s to numbers such that you always have three digits

The code is pasted below. One really fun (not) thing I found out: VBA (or just OO.o’s implementation of it) doesn’t really do type checking. As a result, if r is a string instead of an integer (which I had forgotten), the following code will always execute as True:

If r < 10 Then

Anyways, here is the code, in case it helps anyone at some point:

REM  *****  BASIC  *****

    End If
  Next
End Sub

The latest perpetrator of bile and venom is one Antal István Miklós. This blathering idiot Web Developer called me out personally on his Great Blog (yes, that’s really what it’s called, if you want to see his post), and turned my well-received talk at the KDE 4.0 Release Event into complaint-a-thon.

I’m not going to pick apart all of the technical wrongheadedness he portrays with MySQL/e, Akonadi, and the capabilities therein — Nikolaj posted a nice response to the blog, and the guy would be very well served to *fully* read my posting about mysqle (as well as many of the comments). Much of the blog can also easily be decoded as baseless, factless trolling ("amarok1.x is the slowest KDE3 program, if not it’s surely in the top 3 slowest KDE3 programs"), and the-developers-don’t-agree-with-me-so-they’re-wrong syndrome. But I *am* going to defend my statements at the Release Event.

Let me explain, up front, the format of my Release Event talk. I presented a short introduction to Amarok, for those that did not know it. I then stated some drawbacks we found with the KDE3 platform. With these drawbacks, plus a desire to go cross-platform like many other media players (VNC, Banshee, iTunes, etc.), we had considered the possibility of switching to a Qt-only architecture. But then, KDE4 comes along, with elegant solutions to our problems — Phonon, Solid, Plasma, targeting multiple platforms, and more. Boom — the thoughts of Qt-only go out of our heads, and we commit ourselves fully to KDE4.

Far from a long series of complaints, it’s a success story, showing how the benefits the KDE4 platform offer to us solved our problems, and how they could solve yours too. Apparently, however, it simply shows — from Antal’s blog post title, and probably because he hasn’t bothered to watch past five minutes in — how we just don’t get KDE4.

This may not be apparent to everyone, but Amarok was an early poster child for adoption of many of the Pillars of KDE. We are the only application, to date, that has embedded Plasma inside of our application (with our developers doing a large amount of work to make that possible). (Update: we are technically the first, outside of the plasma workspace, but there are others playing with that now.) Device detection completely relies on Solid (which is one reason Mac and Windows ports have no device support right now). And we have completely standardized on Phonon for our media engine. We’ve also had Oxygen team members working on our icons and our interface.

It’s hard to imagine ways for us to more fully integrate with KDE4 than what we are doing. We’ve gone for KDE4 whole-hog, and it’s ludicrous to suggest otherwise. Picking out random Pillars that we don’t fully integrate with (yet) does not mean that we are not KDE4-oriented. After all, right now we don’t have a use for Marble (who knows? that could change) — but does that mean we don’t get KDE4?

it just reminds me of one of the KDE4.0 release event, where a KDE dev complained that how KDE3 sucked, because they couldn’t port Amarok to Windows, and KHTML had bad performance

It’d be nice for Antal to realize that there is a difference between complaining, and listing drawbacks of a platform. (This doesn’t really fit into how trolls work, of course.) Yes, I said that two of KDE3′s drawbacks were that it made it impossible to port Amarok to Windows (and Mac) and that KHTML rendering was found to be slow. No, I did not say that KDE3 sucked.

But there’s nothing new here. It’s not like Amarok was alone in wanting to port to other platforms. The Release Event had showcases of KDE4 applications running on both Windows and Mac. But Antal has this fixation that Windows and Mac are suddenly all we care about, taking an out-of-context "consider the majority" statement someone (he doesn’t say who) made on IRC about some topic (he doesn’t say what, only that it’s vaguely somehow about performance):

Using mysqle mostly benefits non-KDE4 desktops, because as I said earlier KDE4 will probably have a mysql server anyway, but isn’t improving the KDE4 user experiance top release priority anymore? Is amarok on Windows on Mac more important than getting the best out of amarok on KDE4?

[then, later]

What did the people in the IRC channel had to say about this?

…….

My favorite quote from here is: "consider the majority"

It’s like saying: "consider the majority, which are Windows and Mac users, and screw the KDE4 users"

I think Antal fails to realize that KDE is not just a desktop. Windows and Mac users that might be using Amarok are going to be using *lots* of KDE technologies in the process. Regardless of his mistake, there is certainly no evidence that Amarok cares more about Windows and Mac users, or thinks them the majority of our users. Speaking as a developer, I can tell you that the exact opposite is true.

Antal also clearly doesn’t realize that Akonadi is not a requirement of KDE to run (even if it’s installed), and therefore the best Amarok could do would be to integrate with Akonadi, but not to depend or rely upon it. Maybe he kind of gets it when he says, my emphasis, "KDE4 will probably have a mysql server anyway" — we can’t rely on probably, or maybe. We need to use what works, always.

He even contradicts himself:

He begins with complaining, how slow was rendering amarok’s context with KHTML, so it looks like performance matters in amarok, not that anyone forced them to use KHTML for rendering context…

Make up your mind, Antal. You’re right that no one forced us to use KHTML for rendering context, although WebKit wasn’t available back then, and hooking into Mozilla was a non-starter. But you want us to be integrating with other KDE technologies…right?

One last point:

Jeff Mitchell the developer who spoke at the event that I was referring to, referenced KDE as a family, but where is the love now? The lack of communication between Amarok and the rest of KDE4(Akonandi) doens’t seem to back up Amarok as being a family member.

It is not surprising, given how little he understands of Amarok, KDE, and the integration thereof, that he thinks both that there is a lack of communication between Amarok and the rest of KDE4, and that he implies that Akonadi is the entire rest of KDE4.

I’ve covered a small fraction of the untruths and inaccuracies in his post, but it’s enough — I’ve made my points. I love KDE, I have not publicly disparaged it, we Amarok developers are fully committed to the platform, and we are not putting the Windows and Mac ports at a higher priority than the *nix base.

Any comments will be read, but I may decide not to post them.

I want to point out first that I said that MySQL is going to be Amarok’s only SQL backend. A2′s collection system is very powerful. Just take a look at how varied music sources from Shoutcast, Jamendo, Magnatune, Ampache, MP3Tunes, as well as local sources like iPods and your local file system, are treated as equals in A2. A collection is a collection, and is limited only by what capabilities it advertises it can support (and of course, it can supply its own custom capabilities). It’s not currently enabled, I don’t think, but there’s a Nepomuk-based collection option too. So take heart — this change only affects Amarok’s internal SQL collection, and not other sources (although those sources can store information in the SQL database if they wish to cache information).

Since I mentioned Nepomuk, it’s time to discuss another common question/demand/complaint: KDE has this nice Strigi-Nepomuk thing going on…why aren’t we using it for scanning music and storing information? There are a couple main reasons. The first is that Strigi and Nepomuk are optional, not required. (Update: Strigi is required, but Soprano isn’t, so Nepomuk as a whole is still optional.) We can’t rely on the user installing them, and even if they are installed, we can’t rely on the user to configure them properly (remember that we’re going cross-platform, making it even less likely). The second reason is speed: Amarok’s custom collection scanner is extremely fast and pulls out specific pieces of information with TagLib. Strigi is, by comparison, very slow (it calculates hashes of all files, which means it needs to read the entire file) and pulls out less information. (Update: According to the Strigi developer, and despite what is said on kde-apps.org, Wikipedia, and even the author’s own home page, it does not calculate hashes by default. So it’s possible that Strigi, if properly configured, could be as fast as Amarok’s internal scanner, although whether it would pull out all necessary information, I don’t know. If it’s configured to calculate SHA1 hashes of all files, then it will indeed be far slower.) On a local hard drive, it may not be a big issue, but it sure is a huge issue when you throw networked storage into the picture, which is a very common scenario. I’ve also heard, though don’t remember specifics, that querying and such through Nepomuk is rather slow, compared to a normal SQL database.  Regardless, though, remember that when the Nepomuk-based collection is finished, tracks sourced through a Nepomuk-based collection will have their metadata changes saved back to Nepomuk. So, it’s not that the SQL collection is in place of Nepomuk — they are entirely independent. (Update: I forgot to mention that a Nepomuk collection already exists. It was developed by a GSoCer over the summer. I’m not sure what its status is as far as making the 2.0 release, but we Amarokers both like Strigi/Nepomuk and are excited about the idea of opening up the app and having all your music available right then and there with no pre-configuration. But there is a place for the SQL collection too. As I said: they are complimentary technologies.)

With those topics out of the way, on to the meat.

First, it is important to understand an important pair of facts. Number one: we are not database guys. Sure, we can store data in them, and more or less come up with a working schema, but none of us are gurus/wizards/jedis/etc. This leads in to number two: maintaining three databases was driving us crazy. Every time a minor schema change was needed, it had to be coded up for all three types of databases. Modifying a schema could be trivial for one database type, and super difficult (or impossible) for another. People would report bugs that we couldn’t reproduce, only to find out that it was because we didn’t quite understand how one database or another behaved (or in some cases, none of the active devs were using that type). And so on. So from the beginning of A2 development (and in our fantasies during A1 development) we knew we wanted just one database.

(We did actually look at abstraction layers like QtSQL and others. I’m not going to comment on them much, as I didn’t do the evaluation, but in general they were found to not be flexible enough to handle all of our needs without doing some custom SQL coding (especially in the cases of things like schema changes), which kind of defeats the point. If you want to know more/want to insist that they are, try asking eean, as I think he did the evaluations.)

Now we had to choose the type. At first, SQLite seemed like a good choice. Using transactions, it’s decently fast. It’s pretty stable (those that complain about odd MySQL bugs should talk to markey, as he, being the SQLite maintainer in 1.4, can attest that SQLite’s had its fair share). However, there were a few problems that in the end knocked it out of the running. The first problem is performance. Although for people with small collections it performs fairly well, people with large collections that switched to the MySQL or PostgreSQL backends in A1 would report enormous speed gains when operations performing complex or many queries were performed, such as adding many entries to the playlist, scanning files, or filtering/searching in the collection. Since we want to accommodate users with large collections just as well as those with smaller collections, and since digital music collections aren’t getting smaller, the speed increase for our users with large collections was quite important. Many of our developers, after the switch to mysqle (as we call it, though that’s not the official name), have noticed huge speed increases in their day-to-day use of A2, so that speed increase is carrying through to the embedded server as well as the normal server. That was the first knock against SQLite.

The other blow for SQLite came for a totally different reason. Many users (myself included) have multiple computers sharing a single Amarok database. Assuming all the computers have access to the music at the same mount point (and a few other things are configured right), this allows you to scan once, play everywhere, update the same ratings no matter where you play it, and more. Even if your aren’t sharing the database among multiple computers, many users want their database stored on a particular server for speed, security, or backup reasons. If you think either of these isn’t a common use-case, you’d be quite wrong. MySQL and PostrgreSQL were quite happy with this workload. It’s a total no-go for SQLite, simply because it’s designed for a different purpose. So SQLite had two big knocks against it. K.O.

However, just as we can’t rely on the user to set up Strigi/Nepomuk correctly, we can’t rely on them to get their tables set up in MySQL or PostgreSQL. So we needed the database to be embeddable, so that it could just work for the user without any setup necessary on their part. MySQL, with libmysqld, had the seeds of this in the 4.1 series, it works decently in 5.0, and it’s becoming fully supported (AFAIK) in 5.1. PostgreSQL, on the other hand, does not have any such thing. (They have an interesting and cool concept of their own of embedded SQL though. Update: apparently that is part of the SQL standard. Still pretty cool. Still totally different from what we mean when we are talking about an embedded server.)

So this leaves us with — as you guessed — MySQL. It may not be any particular person’s favorite database (although it is for plenty), and I don’t know how much overhead it *really* has in embedded form, but it fit the bill. It’s both embeddable and can run standalone on the local or a separate machine (yes, this is not supported yet in A2, but it will be). It is fast and robust for large collections. It is well understood by the development team. And most of all, it is a single-backend solution that fills all of our needs.

If you’re still unhappy about our decision, I’m sorry. We try to please most and can’t please everyone. But we’re the ones that develop and support this thing, and so we made a decision based both upon our needs as developers and the real-world use-cases from the collective feedback of thousands of users that have contacted us over the last few years. Please remember that even if most of the comments on the Dot, or to this post, (i.e. much of the sudden visible feedback) are from people that are unhappy with our decision, it is a decision that will actually suit the vast, vast majority of our users better than the other options we currently have.

We’re a project that is known for being good to our users — we listen to them, we try to implement features they want, try to be responsive with support. It’s one of the things that got us where we are today. So please, dear readers — put some faith in us. This has not been an easy decision — we’ve discussed, we’ve argued, we’ve thrown things, we’ve made up, we’ve had an after-the-make-up orgy or two — but in the end it’s what we collectively felt was the right way to go, and we feel that, in the long run, it will make Amarok even mores awesomer. Hopefully you’ll feel that way too.

"We can rebuild him. We have the technology. Better than before. Better, stronger, faster."

A little-known feature in Amarok 1, starting at about 1.4.3, was what was known as Amarok File Tracking, or AFT.  For every single file in your collection, on scan, a unique identifier (UID) was generated from some of the file’s attributes.  If you moved your tracks around your folders, as the incremental scan kicked in, the UID would allow for the file to be identified, and integration throughout Amarok would mean that your statistics, your cached lyrics, and the current playlist would all be updated with the new path.  No longer did you have to worry that moving around your files would mean losing years of statistics.  Or losing your files.

But I’m getting ahead of myself.

See, AFT wasn’t born AFT.  AFT could not track both a file metadata change and a file location change at once, because the UID was being based on file properties such as file length, plus a portion of the file itself hashed together. So you could still lose track of your files.  This was a limitation that was known in advance.

It was also a limitation that didn’t originally exist.  As I said, AFT wasn’t born AFT.  It was born as Advanced Tag Features, or ATF.  ATF was the same idea, but a little different — it would store the generated UID directly in the file’s metadata.  This allowed for superb file tracking capabilities, because unlike generating a UID from a part of a file, if that part of the file changed, you’d still have your UID.  In fact, the only way you *couldn’t* track your file was if you either removed the file’s tag entirely (or some other program removed the UID when it shouldn’t), or if you removed the corresponding information from Amarok’s database. (There are some downsides to this scheme: only certain file types are supported, for instance, determined by the kind of tag they use and the tag’s ability to store this kind of information.)

So why the change?  Well, ATF had a problem, which was related to the structure of Amarok itself, and Amarok’s historical penchant for crashing (which got much better as the 1.4 series progressed).  The outcome is possibly worthy of an entry in The Daily WTF.  In gory detail, here’s the problem.

   1. Amarok would start a collection scan.  The collection scanner was the entity responsible for adding the UIDs to the file metadata.  Important note: the collection scan was a separate process.
   2. Amarok would crash, leaving the collection scan running, although not communicating with anything.  This scanner could be very slow if it was adding the UIDs, depending on whether padding had to be added to the file’s tag.  If this was the case, the entire file would have to be rewritten.
   3. Amarok would be restarted by the user.  Another collection scan process would start.  Becuase UIDs would already exist for the early files, it would very quickly catch up to the first collection scan process.
   4. You now had two collection scan processes generating and writing UIDs at the same time to the same file.  If you were lucky, this would mess up your tag.  If you were unlucky, this toasted your entire file.
   5. Repeat step 4 for the rest of the scan.

ATF was never released in this state, but it did get turned on in SVN.  And a few unlucky users had far too many files end up corrupted, depending on how crashy things became for them.  After we finally realized what the issue was, a user came forward on the mailing list (still trying to find the exact mail or user) proposing a solution that I believe they’d seen in a class.  Essentially, the solution relies on modifications to temporary, uniquely named files instead of the original file, using MD5 checksums to find out of the original file has changed while writing the new file, then using filesystem atomicity guarantees to move the new file back over the old one.  This became the MetaBundleSaver, and it worked quite well, but it was also extremely slow compared to a normal scan.  And most importantly, no one was quite trusting of the whole ATF scheme any more.

So, ATF was renamed to AFT and with it came a new algorithm that wouldn’t touch anyone’s files, but couldn’t track as well.

A couple weeks ago, I added AFT to Amarok 2′s SqlCollection.  Enjoy, everyone — statistics, lyrics, and the playlist are already supported, with support for stored playlists coming eventually.  But there’s more.

Fast forward to today (okay, two days ago).  I’m taking a shower — Wade does insist that there’s something about showers and KDE coders — and I had a thought, which was essentially: there’s absolutely no reason why Amarok 2 can’t use a UID inside a file, if one exists, for superior tracking, and if not, generate a read-only type for normal tracking.

So I created a utility that is built and installed with Amarok 2.  It’s called amarok_afttagger, and it will write UIDs into your files, using a class ported from MetaBundleSaver and called SafeFileSaver to ensure that files are not overwritten/interleaved, even if you run the process twice or three times at once.  It optionally supports recursion if you want to pass in directories, and it can also remove UIDs from your files if you like.  Right now it supports MP3s only, but Vorbis and FLAC support will be coming soon.

I’ve tested it extensively.  I’ve added UIDs to files, removed them from files, regenerated the ones in files, over and over, and still everything is cherry.  And Amarok 2, when it finds these files, can do some awesomely robust file tracking.

I encourage people to give it a run on their MP3s and check it out — if you’re worried by all the Dark Ages info up above and don’t have faith in the implemented solution, back up your files first, or operate on a copy of them, until you’re satisfied it won’t harm your files.  And if you still don’t want to do it, you can enjoy the less awesome but still awesome power of the non-embedded UID file tracking.

Now, I promised this would talk about a possible KDE library.  I’ll eventually be submitting the SafeFileSaver class for hopeful inclusion into kdelibs, so that any application that is worried about data integrity and needs to write to a user’s files can take advantage of it.  It’s very simple to use — you simply give it a file path, and then operate on the file path that’s returned to you when you call prepareToSave(), instead of the original one.  When you’re all done, you call doSave() and it will perform the necessary functions.  That’s it.

Hope this has been enjoyable, and enjoy AFT in Amarok 2.  Play with it and be amazed.  Use amarok_afttagger on your files and be even more amazed.  More information is available here: http://amarok.kde.org/wiki/AFT

So iPod support should be pretty well set by this next weekend, and the next thing on my list is support for MTP devices.

Now, on to the subject line — the PUD now contains an average of 400% more droppiness. That’s because you can now drop on the text of an item instead of just the icon, which means a far larger drop target (and apparently this is how people expected it to behave in the first place).

Next on the agenda is to have text change color on hover so you’re sure of what you’re dropping onto. Complete with a nice looking fade, of course…

]]> http://jefferai.org/2008/06/popup-dropper-now-with-400-more-droppinness/feed/ 5 http://jefferai.org/2008/06/introducing-the-popup-dropper/ http://jefferai.org/2008/06/introducing-the-popup-dropper/#comments Wed, 18 Jun 2008 23:15:20 +0000 jefferai http://jefferai.org/?p=1035 About a year ago we had an issue in Amarok 2 development because we wanted to be able to move portable device track access into collections (this is happening with Alejandro’s GSoC project), but we couldn’t figure out a good paradigm for accessing the various device specific functions. There were some other concerns too, about the right-click menus getting too overloaded, making it hard to find common functions. Finally, when people wanted to drag music to the playlist, they now had to cross the entire window, whereas before it was a quick drag and drop…a lot less mouse movement, and possibly the difference between picking the mouse up and not.

So I had a brainstorm. We have this ginormous context view area in the middle of A2. What if we could put it to good use, and allow it to act as a surrogate menu? For many people, dragging the mouse a bit to drop items on a big target is much easier than navigating small right-click menus, where it’s easy to overshoot your destination and accidentally close submenus. Thus the PUD — the PopUp Dropper — was born. My first sketch of it is here:

(Here’s the preceding page that talks about collection groups. If you like that idea, be sure to let us know at ama...@kde.org.)

The PUD actually lived for a long time in the Amarok source tree, but there were problems. One was that it tied in too tightly to the existing Amarok context view code — something that, as that code was refactored over and over again, was painfully obvious would not work. The other was that the actual popping up and drawing was painfully slow. It seems drawing a QGraphicsView on top of another one wasn’t making Qt happy. Fortunately, as of Qt 4.4, this problem has been solved.

The final nail in the coffin (related to the first one actually) is that some of the people that saw the work wanted to see it available for all of KDE to use. This approach wasn’t going to work.

When Qt 4.4 hit qt-copy, I started to play around with things again. Soon enough I was addicted to making it work. I took the Drag ‘n Drop Robot example and started modifying it to use the PUD library I was making. The goals: make it Qt-only, make it very customizable so it could work in any situation, and make it actually able to replace a right-click menu.

I’m happy to say that those goals have been met. Here’s a list of current features:

• Runs on top of any QWidget (theoretically, though not fully tested).
• Two ways of creating submenus: either call addOverlay(), which essentially gives you a "clean slate" that you start afresh, and that runs on top of the current menu when show() is called, or you create a second PUD and pass it into addSubmenu() on the first, which is quite similar to the way a QMenu would be constructed.
• Can pop or fade in/out with configurable fade timing on a per-submenu basis.
• Background color and transparency, as well as text, is configurable on a per-submenu basis.
• When the mouse leaves the PUD area, it disappears (or the top layer disappears, and keeping it outside makes subsequent layers disappear)…configurable.
• The PUD items contain a PUD action, allowing for both drops (which calls triggered()) and hovers (which calls hovered())…each one individually configurable, of course.
• The items are represented by a SVG element you pass in, along with a renderer (or the default is to use the PUD’s renderer).
• The items arrange themselves automatically (submenu offset is configurable).
• Many more things. All of them configurable (sense a trend?).

I really wanted to make a video available, but I could get neither xvidcap, nor vnc2swf, nor captury working at all. recordmydesktop worked, but interfered with the drops, so it wouldn’t work as intended. Instead, I’m making an "alpha" version of the PUD available for people to see (I had been using a local git repository, which is included in the tarball). Don’t worry…despite me calling it alpha, it’s quite stable (in fact, as you’ll see, it’s already used in Amarok). No dependencies are needed, just qmake the dragdroprobot.pro file, make, and run the dragdroprobot file. Caveats: it’s not commented (see coloritem.cpp for usage examples), I know of a few bugs, it’s not finished (I have several more things to add to it), there is at least one crash that I can’t reproduce reliably and have not found, there are some things I need to test but have so far been unable to (such as more than one submenu layer), and note the copyrights and distribution requirements on the files — it’s a modified example from Qt. Preferably run in a Qt 4.4 environment (i.e. in a KDE trunk login) to get all the benefits of Alien, or else it might be rather slow depending on your system. But with that all said, let me know how you like it.

Finally, it’s been updated in the Amarok source tree. Now that I got the submenu support working, Nikolaj has gotten it integrated. In recent builds of Amarok you can simply start dragging an item from your collection and see it in motion. Some screenshots follow. The first is showing the normal menu, the second shows the submenu that pops up when you hover over the Organize Files item (you can see the previous menu behind it, through the transparency).

So what’s next? Hopefully, this will eventually make it in some way shape or form into kdelibs, assuming the API stays stable (it was written in the kdelibs style, with a d pointer and private classes), making it easy for KDE apps to take advantage of it without being an external dependency. And as it gets completed I’ll be making it available separately for any Qt app out there that wants to use it (not sure what license it will be under…either GPL or BSD). For now, if you want to see the latest versions, keep your eye on the Amarok source code in the src/popupdropper directory (it has a few modifications to my "master" source simply to make it build in the Amarok source tree…specifically, adding moc file includes since it’s not using qmake, and exporting PopupDropperAction).

Comments appreciated!

]]> http://jefferai.org/2008/06/introducing-the-popup-dropper/feed/ 20 http://jefferai.org/2008/04/amarok-soc-media-devices-awesome-ipod-support/ http://jefferai.org/2008/04/amarok-soc-media-devices-awesome-ipod-support/#comments Mon, 28 Apr 2008 14:52:54 +0000 jefferai http://jefferai.org/?p=1034 Introducing Alejandro Wainzinger (xevix on IRC), who is going to be working on media device support in Amarok for SoC 2008:

My name is Alejandro Wainzinger, and I’m going for a Computer Science B.S. at the University of California Santa Cruz, USA. This summer, I’ll be bringing back media device support to Amarok for Apple iPods, MTP and generic devices, and making them fly. I chose this project because I own an iPod, and got frustrated with the speed of loading an iPod with a large database, sync’ing of songs/playlists and album art, unlogged crashes after trying to put a 10,000 song queue onto the iPod, and slightly unreliable iPod model detection. That said, I loved having iPod capability in Amarok, and I couldn’t see Amarok 2 without media device support.

Alejandro is going to work on getting normal functionality for all three types of devices, and then really taking the iPod plugin to town.

Rumor has it he’s then going to attempt making collections out of the devices…

Another reason for the long delay was a partially failed experiment that aimed to provide support for many different media devices via a centralized kioslave. It’s not that this couldn’t still happen, at some point, but it becomes extremely difficult to map protocols that have no notion of filesystems into a filesystem, and have filesystem clients behave properly. There was another foray into a different system that also ended badly — mainly because there was no way I’d get it finished within a year, much less within a few months.

In the end it was decided that Amarok’s device plugin system is really pretty decent, providing relatively bug-free and easy device management for almost every device on the market…and if it ain’t broke, don’t fix it. Of course, it’s using tons of deprecated classes and methods, but as Trolltech says (my emphasis), "we recommend against using these classes in new code." Good point. So let’s get stuff working, and then we can try to design a new, better system in parallel — but at least something will work in the meantime.

So with that decision made, I worked on integrating with Solid. I’m happy to report that today, I plugged in my Creative Zen MicroPhoto, and it was instantly detected and the correct plugin selected. All I had to do was hit Connect. Screenshots:

The first screenshot shows what the media browser looks like with nothing plugged in. At that point, I plugged in my Creative Zen MicroPhoto, and you can see that it was detected and added to the device selector drop-down box:

After this, I hit the Connect button (it’s the one on the top left) and artists were displayed. I hit the custom button just so you could see some of the details available:

So it’s coming along. I expect this device detection to work on pretty much any kind of music player (except generic/vfat/UMS…more on that on a later post, when I have time to work on it), provided that:

1. hal is installed and working.
2. You are using a recent version of libmtp/libnjb for those devices. I’ve also worked with libkarma and libifp but have been unable to actually check them out to see how they’re working since I don’t have those devices. We’re probably going to start a "send us your old media devices" thing going, so that I can work on all of them. libgpod was more hesitant about working with me, but there’s a problem there anyways, as apparently you can’t detect from the USB interface the iPod exposes what kind of iPod it actually is, without connecting to it. Feel free to blame Apple on this one, as it’s probably on purpose in an attempt to keep you on iTunes. So there’s a generic definition already in hal which should be (hopefully, pending testing, send me your old iPods) good enough to at least have it show up as an iPod. Once you click the connect button the library should be able to provide more info.
3. Your distribution’s package of the device access libraries installs the hal fdi files provided by the libraries. For some unfathomable reason many (most?) don’t — even my beloved Gentoo. So start bugging your package maintainers early, because if you don’t have these installed, you’re not going to get autodetecting goodness. Some of these may end up in the main HAL distribution (I have repo access so can add them in) but for many it doesn’t make sense as they’re constantly being updated or changed, and it would require changing in two places instead of one. (Also, some libraries generate these automatically, instead of having a fixed file).

So that’s it for now. The next step is to get manual adding of devices working again. And after that, dealing with generic devices in a smart way (the way will be Banshee-compatible too…thanks to Aaron Bockover for the excellent idea!) In the meantime, watch out for the Amarok Device Donation Program.

]]> http://jefferai.org/2007/10/go-solid-go/feed/ 11 http://jefferai.org/2007/06/the-real-first-khtml-browser-on-windows/ http://jefferai.org/2007/06/the-real-first-khtml-browser-on-windows/#comments Tue, 26 Jun 2007 01:35:03 +0000 jefferai http://jefferai.org/?p=1029 There’s been a lot of hubbub lately about Safari being released on Windows, which is based on WebKit, which is based on KHTML, and how it’ll beat Konqueror as the first KHTML-based browser on Windows.

Then you hear this other camp firing back about Swift, whose homepage proudly declares "The First KHTML Browser for Windows."

However, I’m here to set the record straight. To the best of my knowledge the first KHTML-based browser on Windows is ThunderHawk, which appears to have been using KHTML under-the-hood since at least 2004 and probably at least 2002.

Yes, it’s a WinCE/Windows Mobile browser. But it’s still Windows.

]]> http://jefferai.org/2007/06/the-real-first-khtml-browser-on-windows/feed/ 11 http://jefferai.org/2007/05/device-handling-and-hal/ http://jefferai.org/2007/05/device-handling-and-hal/#comments Sat, 05 May 2007 19:01:35 +0000 jefferai http://jefferai.org/?p=1026 My blog’s been rather inactive for a while. Most of this has been because of being busy, and because of not doing hugely exciting things when I have had Amarok hacking time (mainly some bugfixes in stable branch). But now something’s come along that warrants an entry.

Amarok’s device handling is pretty adequate. Granted it could be better, but we support nearly every device on the market through some plugin or another. But there’s one recurring problem: except in a few cases, Amarok doesn’t know what device you have plugged in, and how to handle it. You have to tell this to Amarok. This means you, and we, have to deal with the manual creation of devices. Well, that’s going to change.

Enter HAL, and its KDE4 paramour Solid. By now, you probably know what these do. HAL, Hardware Abstraction Layer, gathers information about your hardware. Solid reads it (and some other information) and outputs a KDE interface to it. Now by itself, this isn’t enough, because HAL has a namespace called portable_audio_player, but, with a few exceptions, it doesn’t actually say what type of device you’re using — simply "storage" for a USB Mass Storage interface, or "user" meaning that a userspace library needs to handle it.

However, for the last few weeks I’ve been working with HAL developers and the developers of libgpod, libifp, libkarma, libnjb, and libmtp. The goal has been to come up with a way that device access protocols can be auto-detected. If you know what access protocol a device uses, you can load the appropriate plugin automatically. Well, this has been done, and as a result the HAL portable_audio_player namespace spec is being modified to allow libraries to insert information they know about devices into HAL — while at the same time keeping libraries from clobbering each other. Interested parties can find a draft version here, with the final verison going into HAL as soon as the slugs at freedesktop.org set up my account.

After this, I will be adding proper entries in libmtp and libnjb (easy, since I have commit access for both), making entries for libifp and libgpod, and working with the libkarma developers for their entries. After all this is done (probably in the next couple weeks), Amarok 2.0 and other HAL-aware clients should be able to determine and load the correct plugin for just about any device on the market, automatically. Okay…after a lot more hacking on 2.0…but you get the picture.

I’m hoping that when this is done we can remove manual device addition/deletion. It’s a complex part of the code that can be difficult to maintain, and shouldn’t be necessary any longer — if you have a library that can handle the device that Amarok can use, Amarok should already know about the device from HAL. Even generic USB Mass Storage players should be fine, because they should be detected by HAL and exported through Solid as storage volumes. The only thing I can see this affecting negatively is the generic device plugin that just got fixed to support arbitrary KIO paths but I’ve been thinking about pulling that into the Collection some way anyways, now that the Collection can handle many sources, as this would let it handle *lots* of sources

I can’t get database support working yet, but it does build and run. Check it out:

]]> http://jefferai.org/2007/02/amarok-on-mac/feed/ 17 http://jefferai.org/2006/05/bang-bam-boom/ http://jefferai.org/2006/05/bang-bam-boom/#comments Fri, 05 May 2006 22:52:49 +0000 jefferai http://jefferai.org/?p=1024 I’ve been knocking down my to-do list like there’s no tomorrow. I posted recently about the VFAT device rewrite; it is now complete and early testing results seem to be positive (it’s been renamed to "Generic Audio Player" by the way). In addition, ATF backend support should now be fairly complete. I’ve been debugging it for a while and everything seems to work well now. It catches file copies, moves, and renames, and lets you assign new IDs to files through a DCOP call. It doesn’t slow down scanning much either (I’ve been scanning 2082 files in about 30 seconds or so). I have yet to test it on Oggs, but I would imagine that it’d be fairly simple to get that working too.

The next hurdle is actually making amaroK components use ATF. I don’t think that will happen in time for 1.4 final, but in the point releases thereafter hopefully you’ll start to see some components take advantage of the features that ATF provides, to help it really live up to its name.

    The good news is that this time, I’m not hating it as much. This is mainly because I’ve started with a much more hierarchial design that makes tasks that were complex and error-prone in the old device code much more simple and reliable. The end result should be a device plugin that is much more stable and doesn’t have as many weird quirks — with any continuing quirks fixed quickly, of course!

    Today I got done with enough code to display the filesystem and allow for expanding directories. This may not sound like much, but it’s actually a good sign, as these functions seem to work well and stably, and these basic functions being completed means that all the under-the-hood code is working (which was the majority of the rewrite), so the rest of the functions will be easy to implement. And the best news of all is that unlike the old code, I’m very confident in this new code!

]]> http://jefferai.org/2006/05/vfat-rewrite/feed/ 1 http://jefferai.org/2006/04/rip-your-cds-with-kio-and-amarok/ http://jefferai.org/2006/04/rip-your-cds-with-kio-and-amarok/#comments Mon, 24 Apr 2006 07:49:21 +0000 jefferai http://jefferai.org/?p=1019     Yesterday I commited some code to the Collection Browser that allows it to accept URLs with the audiocd:/ protocol.  If your machine supports this (the audiocd:/ kioslave is probably in the kdemultimedia package), you can rip by doing the following;

1. Open KControl.  Go to Sound and Multimedia->Audio CDs.  Configure your ripping preferences.
2. Open Konqueror. Go to audiocd:/
3. Go into whatever folder you want your tracks encoded as, i.e. Ogg Vorbis will contain Vorbis "files" that you can select.
4. Drag these tracks to amaroK’s Collection Browser.

amaroK will perform a KIO copy, which will result in a seamless rip of the track and import of the track into your collection.  Easy as pie, and using KDE’s built-in capabilities to boot!