Amarok
For Shame, “The Linux Critic”
126(Update at the bottom.)
Some people can’t be helped.
Today I saw a post by The Linux Critic. In the post the author, Trent, says “I’m really not a fan of the new Amarok“, which naturally got me curious as to why. After all, if you don’t know what problems people have with your software, it can be hard to improve it.
I had never interacted with Trent or any other member of The Linux Critic before, nor had I ever visited their site. I was there with the best of intentions.
I’m going to let the discussion we had speak for itself. I’ve reproduced it below, because Trent started removing my comments and modifying his own. You can see the original blog post here — it’s not even really relevant to most of the ensuing discussion — however I strongly recommend, in case further modification has been made to the discussion on that page, that you read the reproduction of the discussion below which is accurate as of the time of writing this post. At the beginning Trent was fairly hostile; by the end he was outright nasty.
Me:
What about Amarok 2.3 are you not a fan of?
Trent:
The fact that now it’s just another iTunes clone.
Me:
Um. Right.
I guess I will just assume you’re mixing Amarok up with Banshee or Rhythmbox. No other explanation for it.
Trent:
*sigh*
The purpose of this discussion wasn’t to discuss the problems with newer versions of Amarok. It was to discuss the new music player, Clementine
I guess I will just assume you’re mixing Amarok up with Banshee or Rhythmbox. No other explanation for it.
No. I’ve tried Amarok 2.
I don’t like the layout.
I don’t like the entire UI.
I don’t like that the playlist is too small and in the wrong place.
I don’t like the volume control.
I don’t like that it requires KDE4 to run.
I don’t like the enormous waste of space that is the center panel.
I don’t like the ugly look of the buttons.
I don’t like that I can’t figure out how to do a lot of the things I do in Amarok 1.4.
If it were more like Amarok 1, I’d probably be fine with it.
Me:
Sure, I understand the purpose was to discuss Clementine, but when you said you weren’t a fan of the new Amarok, I was interested in knowing why.
The reasons you give are at least mostly valid — aesthetic differences, which are personal for everyone — as opposed to calling it an iTunes clone, which would be very hard to justify. (I say mostly because it doesn’t require you running KDE4 to run. And if you take issue with it depending on KDE libraries, then it seems odd that this would not be an issue for you with Amarok 1.4, which also depended on KDE libraries.)
Trent:
Amarok 1.4 depended on KDE3 libraries. I’m not opposed to KDE3.
Me:
I don’t know why you’re opposed to KDE4. But it doesn’t really matter; they’re just libraries. I don’t run GNOME, but I don’t get upset when something depends on GTK+. If it’s a useful application, then it’s a useful application.
I then attempted to be helpful, by letting him know that many of his complaints could be addressed if he wished to give Amarok another try:
Me:
FWIW — again, not to go into too much detail, since as you said this post is about Clementine — you can adjust Amarok 2′s entire layout, as well as its playlist layout, however you want.
If you go to the View menu you can unselect the Context View if you don’t want it. You can also unselect Lock Layout and relayout the entire thing — have parts of it top-to-bottom, change the order…you can even drag the components on each other to turn them into tabs, so that your entire view could be the playlist, except for when you want to switch to the Media Sources pane to add music to it.
If you go to the Playlist menu, you can select one of four pre-defined playlist layouts, or customize it to your own layout, including a single line per track with whatever data shown that you want.
Also under View, you may want to check out the Slim toolbar — it has a different volume slider.
This is all based on Amarok 2.3.1; not sure which version(s) you’ve tried.
Trent:
Amarok 2.2 was the last one I tried. One could change a lot of that there too, but after about an hour of futzing around with it I just found myself getting too frustrated with it.
I shouldn’t have to go to those lengths just to get around the goofy design philosophy that went into Amarok 2 (much like the rest of KDE4 for that matter).
It’s like they took a look at Amarok 1.4 and said “Ok, let’s see… how can we break everything about this?”.
Ugh.
Clementine isn’t even close to finished yet, and it’s already more usable out of the box than Amarok 2.
Unless they made some rather massive, sweeping usability changes from 2.2 to 2.3.1, which I doubt.
Me:
I don’t remember all the changes between 2.2 and 2.3, but there are plenty. The Amarok team has always been very responsive to user feedback, and that has driven much of our development in the Amarok 2 series just as it did with Amarok 1.
Regardless of you considering the design philosophy of Amarok 2 to be goofy, the entire point of Amarok was always to provide contextual information to your music — Rediscover Your Music. Amarok 2′s goal was to take that further, by allowing for a context space that didn’t have to compete with your collection browsing, so you can still e.g. see lyrics to the song that’s playing while you browse your collection for the next track.
These days Amarok 2 is pretty much feature-complete w.r.t. 1.4 features, and has huge numbers of features never in Amarok 1. It’s always interesting to me how if you look at Amarok 1.1 vs. 1.4 you see *huge* differences, and people tried out each release and gave feedback which made things even better. With Amarok 2 some people seem to think that it’s static, and never changing, and never improving — so they, say, try Amarok 2.1, don’t like how it works or miss some feature that hadn’t been ported yet, and don’t bother trying 2.3, regardless of the fact that in the same series of point releases in the Amarok 1 days they would have seen massive changes. So instead of providing useful feedback and helping Amarok grow and improve, which it still continues to do quite rapidly, they simply turn venemous about it, often complaining about things that have long been fixed or capabilities that have long since been added.
Fortunately, not everyone is like that, and we still have a great user base that provides feedback that we try to address. For instance, the ability to change layouts and make the playlist hugely customizable was the direct result of feedback from Amarok 2 users that wanted to be able to make it behave similarly to Amarok 1.4, which it can do to a very large extent.
Trent:
So instead of providing useful feedback and helping Amarok grow and improve, which it still continues to do quite rapidly, they simply turn venemous about it, often complaining about things that have long been fixed or capabilities that have long since been added.
I’ve seen the KDE forums and I’ve seen enough of the attacks on users who try to provide feedback. It was enough to tell me that if a person isn’t gushing with love over it, a person is therefore just “resistant to change” or “doesn’t get it”.
Most of the things I label as bugs or design flaws are pointed out time and time again as intentional in KDE4 and related applications.
The philosophy itself I find repugnant. That’s why I left KDE completely after trying KDE 4.0, 4.1, 4.2, and 4.3.
While I’m sure Amarok 2.3.1 might be different and maybe even flexible enough to emulate Amarok 1.4, I’ve found that every time I’ve tried anything KDE related in the last 2 years I get frustrated, angry, and venomous.
So no, I haven’t been forthcoming with feedback. I never even know where to begin. There are so many things wrong with the directions KDE4 and Amarok devs have gone, I find myself overwhelmed any time I try to enumerate them.
And again, this discussion is about Clementine, not Amarok 2.3.1. Amarok 2.3.1 might be awesome, that’s great. For people that don’t mind the layout, and the difficult UI, and all the other stuff I simply can’t stand, that’s fine by me. They can use it.
But as Jules said in Pulp Fiction, “Sewer rat might taste like pumpkin pie, but I wouldn’t know, because I wouldn’t eat the filthy m**********r.”
Fortunately, not everyone is like that, and we still have a great user base that provides feedback that we try to address.
Ah. Gotcha. “We”. Meaning that you’re one of the Amarok developers that ruined my favorite music player.
Well, that explains why you’re trolling my blog post.
Me:
Yes. Trolling. Asking for details as to a comment you made in your blog post, and then trying to provide helpful information based on your feedback. Right.
I guess I shouldn’t be surprised. From your comment, you clearly are determined to put down KDE4 and anything related to it, and attempts to be helpful are unwanted.
This is the point where Trent starts modifying history. This was his original reply:
Considering that I don’t use KDE4, I’m not sure that “attempts to be helpful” are even relevant.
To which I replied:
You listed complaints about Amarok 2; I explained how they could be addressed. That’s attempting to be helpful, and relevant to the discussion.
At this point Trent deleted my comment above, and went through two revisions of his previous comment. The first one is below:
Considering that I don’t use KDE4, I’m not sure that “attempts to be helpful” are even relevant.
As an FYI to any other KDE developers or fans that might be reading: If you have something useful to contribute to any discussion on The Linux Critic, your comments are welcome.
If all you are going to do is pick fights about why EVERYBODY can’t love every little thing KDE4 and its apps, your comments are not.
This has happened again and again, and I’m not going to tolerate it any further.
And here is his second revision, which is still the latest at the time of publishing this post:
Considering that I don’t use KDE4, I’m not sure that “attempts to be helpful” are even relevant.
As an FYI to any other KDE developers or fans that might be reading: If you have something useful to contribute to any discussion on The Linux Critic, your comments are welcome.
If all you are going to do is pick fights about why EVERYBODY can’t love every little thing about KDE4 and its apps, your comments are not.
This has happened again and again, and I’m not going to tolerate it any further. Play nice, or play somewhere else.
Thanks.
So there you have it. Not much else to say. I’d never heard of The Linux Critic before, and now I know why.
Update: Trent replied with more accusations of trolling (does he even know what that word means, considering he’s the one trolling on KDE4?), of purposefully trying to derail a discussion about Clementine (what discussion?), of some sort of KDE4 lovefest, and all sorts of other manners of odd accusations against a person he’d never met before today and the discussion above. This guy is off his rocker. Here’s the comment:
And I see my KDE troll has gone off to sulk.
Just as a heads-up, trying to derail a discussion about the Clementine music player and turning it into all about KDE4 is not “trying to be helpful”, and it is not “relevant”. It’s rude, insulting, and I consider it to be trolling.
While I don’t typically behave like a topic nazi here with respect to keeping every single comment on-topic, when it comes to KDE4 trolls, I have very little patience, because those comments always end up going the same places, and it’s never constructive, always just fight-picking.
In this case, I’ll freely admit at least some culpability, because I responded to the initial question about Amarok, despite my better judgement.
Here’s a hint: not everybody loves KDE4. Not everybody wants to use it. Some of us prefer not to use KDE4, and some of us (gasp!) even prefer the KDE3 versions of those apps!
Baffling, I know. But it’s true. That’s why I made this post yesterday about Clementine. There are lots of other people out there that used to love Amarok 1, and don’t want Amarok 2.
For those people, seeing a new, stable, usable fork of Amarok 1.4 is a really great and exciting thing, and I thought that others should know about it. It’s very positive news when for a lot of us, the past few years have been spent trying in vain to find a replacement for that now dead application.
And in my experience, when KDE4 trolls come around, trying to change the discussion into an argument about why????????, it’s not relevant, not helpful, and is anything but useful to the topic at hand.
The reason I’m commenting like this is because I want to clarify that. I’m drawing a line on this subject because I’m tired of this.
To Jeff: those of us who have left the KDE world don’t want your help. You can keep it. Don’t call us. We’ll call you. So scurry off to your KDE4 lovefest and leave the rest of us be to solve the problems you created for us. You’ve done more than enough.
What a swell guy.
Update 2: Here’s Trent’s response to one person who attempted to go onto his blog and tell it like it is (thanks Marand!). One editorial note: Trent claims “you also didn’t see some of the other garbage of his I deleted.” Every tiny bit of my interaction with him is in the original post above. Not one word has been omitted. It’s possible he has since deleted some more of my comments from his blog in order to rewrite history, as I have shown above, but what you see above in the original post is all of the “garbage”.
Marand:
Maybe I’m missing something, but I don’t see how he was trolling. I see a developer trying to communicate with a former user to get useful feedback and being treated with hostility for it.
If you don’t like something, that’s fine. If you can’t explain why you don’t like it, that’s also fine. However, it isn’t his fault you can’t offer a well-reasoned explanation of why you dislike the application, so I think you crossed a line lashing out at him over your inability to express your opinions constructively.
It looks like the discussion fell apart when constructive advice was offered to solve your various complaints. Lacking better arguments, you lashed out. You should have just left it at “I don’t like it because it’s different, and I have an irrational hatred of the number 4″, it would have been more mature.
Here, let me give you a better example to use when saying you dislike Amarok 2. This is a problem that has plagued the app for me in all versions since the change. Frequently (several times a day), I have to restart Amarok 2 because the dynamic playlist breaks and stops updating. Haven’t been able to figure out why, but it seems to be related to the collection scanning. It’s frustrated me enough that I’ve considered changing players, though I like Amarok enough (even 2!) to still use it despite this problem.
For the record, I’m not a developer of KDE, Amarok, or anything else. I use KDE4 and I actually like it, but I don’t care what anyone else uses or likes. My only motivation for this post is I think that, if you like or dislike something, you should either be able to provide clear explanation of why, or you should admit you cannot.
Trent:
Maybe I’m missing something, but I don’t see how he was trolling.
He was, as happens frequently, someone who tried (successfully, I might add) to derail a topic to talk about something else after being repeatedly warned that it was off-topic. While that can pretty easily fit the definition of someone needing their comments moderated, he was doing it in an inflammatory, extraneous manner. You also didn’t see some of the other garbage of his I deleted.
My biggest mistake was in responding to him to begin with, given that his very first comment was off-topic (and I know from experience where those usually end up leading).
If you don’t like something, that’s fine. If you can’t explain why you don’t like it, that’s also fine. However, it isn’t his fault you can’t offer a well-reasoned explanation of why you dislike the application, so I think you crossed a line lashing out at him over your inability to express your opinions constructively.
I’m perfectly capable of explaining why I don’t like Amarok 2. If you bothered reading before you posted, you’d notice that I gave him several valid reasons why I don’t use Amarok 2, despite the fact that this article was about something else, specifically a program that Amarok 1.4 users may not be aware of that I’ve found to be a very positive direction in music player development for a change.And considering that The Linux Critic is my blog — not his, and not yours –, and it’s my prerogative to determine content here, including content of discussions about posts, and considering that his discussion was off-topic and inflammatory, I was in no way “out of line” in lashing out at him. As I’m sure you know, Jeff has his own blog where he went to whine, and note that I didn’t go over there and change the subject to something that he found inappropriate, or spout idiocy at his opinions. He can have his opinions, he can say what he wants.
And he can express them somewhere else. They were inappropriate here. As I mentioned, I made a mistake in responding to him to begin with.
It looks like the discussion fell apart when constructive advice was offered to solve your various complaints.
The discussion “fell apart” when he tried to change the subject from Clementine 0.4 to Amarok 2 and KDE4. After repeated warnings he continued to do it, so I deleted his further attempts at doing so and I blacklisted his IP. I have made it very clear here in the past that I will not tolerate spammers and trolls, and I felt that I was pretty reasonable in the amount of leeway I gave him. Most of the time I just delete those types of comments outright, because they inevitably lead to (surprise!) more off-topic discussion like we’re having right now.
Lacking better arguments, you lashed out.
Ah. So, since I went out of my way to present arguments that were in response to off-topic, inflammatory trolling, and you happen to disagree with my opinion, they’re “lacking better arguments”. Gotcha.
You should have just left it at “I don’t like it because it’s different, and I have an irrational hatred of the number 4″, it would have been more mature.
And you were wondering why I don’t like KDE4 trolls commenting on my blog posts? Again, you apparently missed the several valid reasons I gave why I dislike Amarok 2. And you apparently missed the part where I mentioned that I’ve used KDE 4.0, 4.1, 4.2, and 4.3, and have dismissed it as a valid replacement for KDE 3.5.10.If I didn’t like things because they are “different”, I’d still be using Windows 2000.
And on the subject of maturity, derailing a topic to insult people because you disagree with their opinions, that wins a prize too, sport. Keep it up.
Here, let me give you a better example to use when saying you dislike Amarok 2. This is a problem that has plagued the app for me in all versions since the change. Frequently (several times a day), I have to restart Amarok 2 because the dynamic playlist breaks and stops updating. Haven’t been able to figure out why, but it seems to be related to the collection scanning. It’s frustrated me enough that I’ve considered changing players, though I like Amarok enough (even 2!) to still use it despite this problem.
And the next time I write an article here about the problems with Amarok 2 and how they could be improved, that will be a valid and relevant thing to discuss. But considering that this particular one is about Clementine, that would be a bit out of place here, as I told Jeff several times before I banned him.
I use KDE4 and I actually like it, but I don’t care what anyone else uses or likes.
No, like other KDE4 fanboys I’ve encountered, you obviously can’t stand it when someone doesn’t get all starry-eyed at the desktop environment you love so much, and just have to shoot your mouth off any time someone expresses an opinion to the contrary, whether it’s appropriate for you to do so or not.
My only motivation for this post is I think that, if you like or dislike something, you should either be able to provide clear explanation of why, or you should admit you cannot.
And I did, above, in response to Jeff. Even though it was against my better judgement, and in violation of my own policy against feeding the trolls here as it was off-topic and inflammatory.What I probably should have done was delete his posts and ban him right off the bat. Had I known he was just an Amarok developer coming here to pick a fight, that’s what I would have done.
The biggest problem is, I tend to give people far too much leeway here sometimes, and look what it gets me.
Yes, look what it gets him.
Update 3: Another brilliant post from Trent in the comments of his blog. The real highlight is his characterization of my interaction with him, in which I apparently expressed, using lots of punctuation marks and emphasis, my disbelief that he didn’t like every tiny thing about, apparently, anything. He says “here’s how these almost always go” and finishes by saying “Every. Single. Time.” Which shows that his arguments aren’t even internally consistent.
Trent:
I actually like KDE4
but I’m certainly not going to push it on someone who’s not interested.
There’s nothing wrong with liking KDE4. It’s not for me, that’s all.
I’ve tried engaging with the developers at the Amarok forums, they are not interested in any critiques or comments on the UI. Politeness gets you ignored or patronised, frustration gets you deleted. Unless you’re part of the echo chamber don’t bother.
*nods*I found that most of the issues I had with KDE4 and Amarok, when I looked around where one would submit such feedback, were things that were there by design, and would never be “fixed”, because they weren’t broken. Things like this convinced me it was best to just move on entirely. Despite what any of them said, they weren’t really interested in feedback that didn’t reinforce what they were doing.
They even come here and try to offer “help”.
If they truly wanted to help, they’d have done what the little team working on Clementine has done. Port Amarok 1.4 to Qt4 to take advantage of some of Qt4′s features, and make it worthy of the name, rather than… whatever it is they’ve done with it.
That would have helped. Trolling blog posts about replacement applications doesn’t help at all. It just makes me mad.
I actually don’t mind they have their vision and are somewhat single minded about it – that’s their prerogative as the developers, its this pretence that they are open to feedback that is quite irritating and the tendency to evangelise.
Yeah, that’s exactly what spins me up so much. Here’s how these almost always go.1) Wait, you don’t LIKE *whatever*???? Inconceivable! Why?
2) Well what don’t you like about it?
3) Oh. Well, I think you don’t like it because youdon’t understand it.
4) Oh. In that case then, you don’t like it becauseyou’re resistant to change.
5) Well, I’m going to assume you’re stupid then, since you don’t like it. Here’s how to use it.
6) Why are you getting mad at me? I’M ONLY TRYING TO HELP.
Every. Single. Time.
Meh – Amarok is dead (for me anyway). Long live Clementine. And RhythmBox. And Banshee. And Bangarang … you get the idea
Yeah. Amarok died with 1.4 for me. I’ve moved on, particularly now that it looks like a replacement (in Clementine) is in the works. It might take a while for it to catch up, but I’m pretty hopeful that those of us still using Amarok 1.4 can move on from it soon enough.Thanks for the comment.
Update 4: Valorie Zimmerman wrote a nice reply on Trent’s blog encouraging him to work with others, instead of simply spewing bile. Trent’s idea of responding to such a nice comment is to entirely miss her well-articulated point, insult me some more (to be expected), continue to insist that my comments were inflammatory and “intended to start an argument”, and then command her to “move on”:
Valorie:
You’ve completely misconstrued Jefferai’s response to you. Why the hatefullness?
Really, everyone here agrees on two things: we love music, and we love free and open software. So why not use the player you like, and contribute to that if you can. F/OSS works when people work on the projects they love.
F/OSS is just dragged down by hating on people and projects. Please stop doing that. I love Amarok, but point people to Clementine (or other players) if they want help finding alternatives. No project can meet all needs, or keep everyone happy. Enjoy diversity!
Trent:
I love Amarok, but point people to Clementine (or other players) if they want help finding alternatives. No project can meet all needs, or keep everyone happy. Enjoy diversity!
That was precisely the entire point of this article. To point people to Clementine, and to provide some reasons “why”. I think I did a reasonable job of that, unless you read a different article than I did.
My responses to Jeff were responses to off-topic, inflammatory comments intended to start an argument. I’ve already admitted that I should not have responded to him. It was my fault, I’ll freely admit that. Move on.

Amarok Mobile – The Beginning
4This post is actually rather after the fact. I’ve been super busy, and my blogging tends to fall way, way behind when that happens. But I wanted to tell people about plans for Amarok Mobile, where things stand now, and what needs to be done. (And how you could help!)
First off, I need to give a big shout-out to Nokia. A while back, they donated ten n900 devices to KDE developers who submitted proposals describing what they would work on. I was one of the lucky winners, and this provided both the motivation and means to start working on an Amarok Mobile.
I actually was not initially supposed to use this for Amarok Mobile development — I had sumitted a proposal to bring functionality similar to the beloved* Popup Dropper into a library for Maemo programs. Problem: having never used Maemo before, I didn’t realize it had longpress functionality (my previous phone was an iPhone).
So, not wanting to reinvent the wheel needlessely, I instead decided to start working on exactly what everyone had assumed/thought/wanted me to be working on when they heard I was one of the n900 recipients: Amarok Mobile.
Below is a record of some of the work that has been done, and that remains to be done.
Core work
In large part due to its very rapid pace of development, Amarok’s code base has had a habit of growing haphazardly, and one of the results has been a lot of incestuous code…things coupled when they really shouldn’t be, code being referenced and implemented all over the place, and so on.
As a result, one of the first things I did was to work on separating out code that provide core functionality — things like podcasts, playlists, our metadata system, collections, and more — and boil them down to the basics. The rule for the core library is: no GUI code, and nothing inside core may reference any files outside of core. The point is that core should comprise the essence of Amarok, and much of what’s left are the possibly platform-specific implementations. In addition, I standardized namespaces for the parts of core, standardized include headers (which makes it much easier to handle file moves later, since they are fully-pathed from the top level source directory), and had absolutely countless compilation cycles. It was an absolute fsckload of work.
However, I think it paid off. We now have, inside src/, a core/ directory and a core-impl/ directory. If we studiously keep core/ separated, it will provide an excellent base on which alternative GUIs and implementations of those core classes can be built. In addition, more things will be moved there, or at least the paradigm will continue being followed; for instance, our great contributor Erik Hovland has been making progress on separating out the GUI layer from our Services framework so that we can have the great services like Ampache and Magnatune and Last.fm and so on be relatively easily activated on other platforms.
SQL
Amarok 2 currently uses MySQL exclusively, allowing you to either use a MySQL server or libmysqld for an embedded database. This has pros and cons, but the really relevant pro is the huge speed gains we get over SQLite. (Before people comment about how we must be doing something wrong: we’re not the only project that has seen serious speed problems with SQLite — I know that Quassel has had similar issues; and if you’re convinced that it’s our schema or some other such thing, you’re welcome to help us improve it. Many others have made this claim, and none of them have ever taken our invitation to actually help us improve things.)
So the obvious question comes up: what about Amarok Mobile? I’m not sure how realistic it is to expect that we can get libmysqld running efficiently on the n900, especially given its memory limitations. At the same time, SQLite is readily available. But, we want to avoid writing tons of custom code.
As it turns out Quassel has a really great storage abstraction using QtSql, and they use both SQLite and PostgreSQL. So, if we copied their really nice approach, we could maybe support SQLite with Amarok Mobile while reducing developer effort to a minimum (since none of us are SQL experts).
One of the big problems we’ve had with QtSql before is that the QMYSQL plugin can support libmysqld (except that it can’t; more on that in a second), but in one of the most boneheaded software design choices I’ve ever seen in my life, MySQL determines whether you’re connecting to a server or an embedded server based on which library your program *links* to. This is not a runtime option. Wut.
So — the obvious option is to bring to Qt what we do in Amarok — instead of having a QMYSQL driver that has to link to one of the other, have a QMYSQL driver linking to the server libraries and a QMYSQLE driver linking to the embedded libraries, and let application developers choose one or both. I began working on this, and have been successful in building both drivers, and have high hopes I can get it included into Qt 4.8 (there’s necessarily a lot of code duplication, so I’ll have to see what they say about that…I may be able to abstract some of it into shared functions, but anywhere that mysql in any case was used, I had to replace with mysqle).
Now, I said that it can’t actually support libmysqld. The reason is that you pass in options for the embedded server upon library init — but QtSql *always* initializes the library with null options. Which means, among other things, that it will use /var/lib/mysql (or whatever your compiled-in default is) for its database directory, which normally means that if you’re not the root user, you can’t actually do anything because it can’t create its files. Whoops.
I can’t break ABI or API, so I’ve been wondering what to do about this — some of the Qt guys suggested I create a new SQL framework with these drivers in it, but I don’t think that’s a great option. Instead, I plan on creating a new connection option (see the table here). Normally these are basically mapped to MySQL options and used like this:
db.setConnectOptions("CLIENT_SSL=1;CLIENT_IGNORE_SPACE=1");
This connection option will not be mapped directly to a MySQL option, but will instead take a delimited list of options, something like this:
db.setConnectOptions("CLIENT_SSL=1;CLIENT_IGNORE_SPACE=1;MYSQLD_INIT_OPTIONS=myoption1=myvalue1;;myoption2=myvalue2;;myoption3=myvalue3");
This would let you pass arbitrary options, and this list would be split and each of the options would be passed into library initialization. It’s API/ABI compatible and relatively straightforward, so it’s the best thing I can think of right now.
Future work
The next steps are to finish the QtSql modifications and get those submitted upstream. Maemo right now has Qt 4.6, but *eventually* will have 4.8…so when 4.8 is coming around, if my QtSql changes are in there, Amarok can try migrating to QtSql; to finish services GUI separation and separation of other parts of Amarok (like AmarokURLs), and to begin writing a GUI, ideally using the Qt Kinetic declarative UI library — when you’re starting from scratch, you might as well do it right. Fortunately some of these tasks will be much easier now that Qt is fully supported/included in the proper locations with the n900 PR 1.2 firmware release.
My time has become incredibly limited in the past month, so if you have time and are interested in helping, join us!
* Okay, I know one person that really, really hates it. And a lot of people that never really realized it was there. But a lot like it!
Rescanning Single Folders
7Quite often — or at least, “often enough” — we get people that want to rescan a particular folder in Amarok. This is usually the result of them changing tags and wanting Amarok to pick it up. (Remember, for efficiency reasons Amarok watches the modified times of the collection directories, not the collection files, so adding or deleting (or renaming) files will trigger an incremental scan but modifying a file’s contents won’t.)
Previously, our advice was “touch the folder” or in the worst case “just do a full rescan.” Now there’s slightly better advice we can give. If you right-click on a folder in the collection setup dialog, and if the folder was already in your collection (i.e. you didn’t just check it a moment ago), it will give you the option to rescan that folder.
The text above that field has also been modified slightly to indicate that this is possible. Check it out:
This is in current git master and will be in the 2.2.3 release!
The Collection Scanner’s Ultimate Speed Bump
7A couple of weeks ago I spent a long period of time looking at ways to increase scanning speed. Yes, again. I had written previously (here and here) on changes I’d made to increase the speed of scanning collections, and I’d made a lot of other changes that I didn’t write about. These certainly helped in many situations, and made significant differences for particular users, but overall we still had a large speed boondoggle: the number of queries issued per track to the database.
This number was at least three. At the very minimum, assuming the artist, album, genre, composer, and year had been cached, you had a lookup for the uniqueid, a lookup to check for whether the track was part of a compilation, and an insert. This number could be significantly higher in some situations; but these database queries were a large part of the reason that a scan might run, finish disk I/O, and then spend quite a long time eating up CPU time before it was done.
I looked at increasing the number of values I was pulling in using UNION queries, which was currently up to five values (depending on whether they were already cached). I also looked at prepared statements, which it turns out were not likely to give us a large speed bump and are an absolute and utter pain in the ass when using the C API as we are. (We use the C API for historical reasons; also, the official C++ API is not easily available in many distributions, and there are many unofficial contenders.)
Finally, I looked at ripping the guts out of the scan result processor and doing what several of us devs had decided would probably be the best way to improve speed, but was at the same time the most challenging: replacing the SQL statements with local memory storage, populating these at the beginning of a scan and re-populating the database at the end. It wasn’t easy. I selected data structures very carefully for speed, looking at how often each query would be run and the time complexities of insertion and lookup into various types of structures. In addition, because of a few complex queries using joins, the behavior we needed could only be emulated by using very complex data structures (one of them is a QHash<QPair<int, QString>, QStringList*>; another is a QHash<QString, QLinkedList<QStringList*> *>). I also wanted to minimize memory usage…I did some rough back-of-the-hand calculations which indicated that using my design, for the vast, vast majority of collections, memory usage during the scan was likely to go up by only a couple of megabytes at the very most, which would be reclaimed at the end.
In addition, I implemented logic to drastically decrease the total queries needed for insertion. Since I’m constructing the final queries from the data structures, I could easily do insertion of multiple values at once. So now the database is queried at the start to see the maximum query size it will accept; then values for insertion are appended to the query up until this value is hit, at which point the query is run and a new query begun. By default I think this is one megabyte for most installations, which means that instead of possibly thousands upon thousands of insertion queries (depending on the size of your collection), you’d have to have an absolutely extraordinarily large collection to have more than 25 or so queries for the entire scan. Since there is not only round-trip delay sending data to and receiving responses from the database but also the database must parse each query each time it’s run (the purpose of prepared statements is to reduce this), then this makes a drastic difference. To put it another way, the number of queries per track has gone from a minimum of three to an asymptotic value of zero.
Overall, the work seems to have paid off rather nicely. Benchmark reports that I got back indicated anywhere from 30% to 300% gains in the total time for a scan, depending on size of collection and your particular I/O throughput/CPU/etc.
I call this the Ultimate Speed Bump in the title because there is far less that can be done from this point on to increase speed, at least as far as I can currently tell — this was the Big Mama, the elephant in the room. But hopefully it will increase scan processing enough that further increases won’t really be so much of a necessity anymore.
One other data point: the database is now case-sensitive, which is something we’ve intended to do for a long time but is now being done. It’s been a very long-standing feature request that we’ve finally implemented (we wanted it too), and thankfully it wasn’t much work to do this and change the appropriate points in the rest of the code.
(By the by, this will all be in 2.2.1 — enjoy!)
Speed never gets old. At least, in software.
12As my regular readers — such as that may be, considering how rarely I post — may know, I’ve been doing all sorts of things to speed up scanning your files into Amarok’s local collection. I have some really nice news on that front: a few goodies that will be very useful to you, especially if you have very flat collections. These improvements are in the forthcoming 2.2.1.
First up, and the lesser of the two: If a directory is encountered multiple times during scan, the scanner will now only scan them once. There are a few corner cases where this could happen (for instance, if you have a top-level directory specified as well as one of its subdirectories, and the subdirectory’s name changes, changing both its and its parent’s mtime). Not too useful for most cases, but was implemented while working on…
Second, and the biggie: When doing an incremental recursive scan, Amarok will now no longer scan subdirectories whose mtimes have not changed. This is big news if you have a large, flat collection (which I dislike myself but some really enjoy…to each their own). For instance, in a setup like 3,000 files in your main folder, plus a few thousand files each in a few subfolders for a total of 10,000 tracks, if you added a single file to the top-level folder and had incremental recursive scanning turned on (the default), you’d cause a rescan of your entire music collection — all 10,000 tracks or so. Now, if you add a single file to the top-level folder, you’ll only cause a rescan of 3,000 tracks…which is still a lot, but even more of a reason to use a hierarchy.
This should help scanning time even for those with hierarchies, as adding a new album to an artist won’t cause the artist’s other album folders to be rescanned, or adding a new artist to a genre won’t cause all the artists and albums in that genre to rescan.
It’s really the proper way to do things, but it required some changes to the data sent between Amarok proper and the collection scanner, so had to be done carefully (as far as I am currently aware I didnt’ cause any regressions). Regardless, I’m glad to have it in there and working, and hopefully you will be too.
Say goodbye to history
7And by that, I mean say goodbye to an historical (read: old) bug.
I’d heard these whispers recently about “whenever I add or remove an album from the collection and do an incremental update, my collection gets messed up.” Okay — we’ve heard these whispers for a long time, but there were so many other things that had to be fixed first that it wasn’t clear whether this was a symptom of a different problem or a problem of its own…and it could always be fixed with a full rescan.
However, with the collection being much more solid these days and with this being the only super-visible bug left that I knew of, I decided to tackle this. It turns out that, once the rest of the collection was behaving, this wasn’t that hard to find…it just took a lot of debug tracing, because it wasn’t obvious. Since the cause was the wrong field being used in a DB query to remove some tracks during an incremental scan, with anything other than a very tiny test collection it could quickly start to make no logical sense what was happening.
The reason I say the bug is historical is that the code causing it has been there since May 2008. While admitting that provides some fodder for naysayers and haters to harp on Amarok for such visible and longstanding bugs, I prefer to take the other approach: it means that all the various issues users have found since the total rewrite that was 2.0 *are* being found and *are* being solved. Some of them just take some time; it’s a *lot* of code. But the proof is in the 2.2 ChangeLog.
I managed to sneak in the fix a day before 2.2 was tagged, so when Amarok 2.2 comes out scanning (both full and incremental) should be really quite solid. In fact, since this has been fixed in git, I’ve yet to hear of any more problems, just lots of happy users. It’s just another way that the (very close!) 2.2 is going to *rock*.
AFT and MusicBrainz track identifiers, redux
2A bit ago I blogged about how Amarok File Tracking can now use MusicBrainz identifiers to do its stuff.
Then, a little while later, I started getting bug reports of peoples’ music disappearing from their collection, and requested some of the reporters send me some files. One of the users did so, and I found something curious in his tags (if I had a penny for every time I’ve personally seen users have something odd or strange in their tags, I’d have…well, a few dollars at least). Several of his files had full MusicBrainz tags — with absolutely no data populating them, meaning that the MusicBrainz identifier (and all other MB data) for all of those files was ending up the same (blank) and Amarok was thinking them the same file.
It was a quick fix (use generated non-embedded AFT IDs when the MB tags are empty) but just adds to the evidence that you can never, ever trust users’ tags. Also, that users that use your Git-based version or betas really rock for finding this stuff before release…so in case I don’t say this enough: thanks users!
AFT and MusicBrainz track identifiers
2A heads-up: Amarok File Tracking can now use MusicBrainz track identifiers for its embedded IDs. This means people that have used Picard to tag their files but not amarok_afttagger can still get some embedded AFT goodness! It also enables an interesting "mode" because it essentially enables song tracking vs. actual file tracking (which you may or may not want, depending on your particular needs).
Full details are here.
DB changes — call for benchmarkers!
11I’ve done some work in trunk over the past week that may have a huge impact on many of you Amarokers. Read on, and if you can do some benchmarks for me, fantastic.
First, the schema/table changes.
- We’ve seen some issues where people have, for whatever reason, ended up with InnoDB tables instead of MyISAM tables. This is probably the result of their DB being created long ago before we were explicitly telling the mysqle startup to skip InnoDB. This mainly causes a problem because some columns cannot be as wide as we’d like them to be when using InnoDB. So, the first thing being done is that an ALTER TABLE is being forced on every table to explicitly convert to MyISAM. In addition, ENGINE parameters are now used during table creation to be more explicit in the future.
- Some of you might have seen complaints in the debug output about indexes not being able to be created due to a max key length, which by default in MySQL is 1000 (compile-time option). So, some columns have had their widths adjusted so that all indexes are now successfully created.
Now, the other changes:
As we added more features, scanning got slow. Like, really slow. You’d spend more time running SQL queries than actually scanning your files. So I’ve been aiming to change that.
Over the past week I’ve committed changes that remove, per track, anywhere from 1 to 6 SQL queries. The exact amount is highly dependent on your file set, but there is a minimum of one less SQL query per track. If you’ve done a lot of file moves and AFT kicks in, it’ll be an even more massive speedup. I’m going to try to do some further tuning, but already results are looking positive.
Nikolaj has reported that his scan time went from 68 seconds to 18 seconds — more than 3x faster. Mikko didn’t notice a speedup, but he said that whereas scanning used to peg his CPU at 100%, it no longer does so. What I want to know is: how does this affect *you*?
If you want to help, do the following:
- Backup your DB. If you’re using external MySQL do a mysqldump, if you’re using internal MySQLe backup the mysqle folder in the Amarok data directory.
- Update to a revision from a week ago…say, 995000.
- Wipe your DB.
- Start Amarok — it will do a full scan because of the empty DB. Time it as it does the scan.
- Repeat steps 3 & 4, so that you can see what the time is like after caching.
- Update to current trunk (at least 998470).
- Repeat step 3.
- Repeat steps 4 and 5.
Then leave a reply here with your values. If you watch your CPU during each of the scans, report that here too. Thanks!
AFT fixed on the Playlist
0Yes, another one of my semi-habitual posts about AFT. Just a short one though.
In revision 992942, I finally fixed a bug that has kept AFT working for the playlist in certain situations (although it had previously been working for both saved user playlists and statistics). This means that if you have a track in the playlist, move it to another location, and it is then scanned in that new location (remember, kids, it uses folder mtime to determine whether to scan a folder, so when in doubt do a "touch ."), the track in the playlist should remain valid and play the song in the new location. As the playlist use case was one of the initial reasons for the development of AFT back in Amarok 1.4, you can imagine I’m happy that it’s finally (seeming to be) working again in all scenarios, instead of failing in certain situations.


Recent Comments