Speed never gets old. At least, in software.
As my regular readers -- such as that may be, considering how rarely I post -- may know, I've been doing all sorts of things to speed up scanning your files into Amarok's local collection. I have some really nice news on that front: a few goodies that will be very useful to you, especially if you have very flat collections. These improvements are in the forthcoming 2.2.1.
First up, and the lesser of the two: If a directory is encountered multiple times during scan, the scanner will now only scan them once. There are a few corner cases where this could happen (for instance, if you have a top-level directory specified as well as one of its subdirectories, and the subdirectory's name changes, changing both its and its parent's mtime). Not too useful for most cases, but was implemented while working on...
Second, and the biggie: When doing an incremental recursive scan, Amarok will now no longer scan subdirectories whose mtimes have not changed. This is big news if you have a large, flat collection (which I dislike myself but some really enjoy...to each their own). For instance, in a setup like 3,000 files in your main folder, plus a few thousand files each in a few subfolders for a total of 10,000 tracks, if you added a single file to the top-level folder and had incremental recursive scanning turned on (the default), you'd cause a rescan of your entire music collection -- all 10,000 tracks or so. Now, if you add a single file to the top-level folder, you'll only cause a rescan of 3,000 tracks...which is still a lot, but even more of a reason to use a hierarchy.
This should help scanning time even for those with hierarchies, as adding a new album to an artist won't cause the artist's other album folders to be rescanned, or adding a new artist to a genre won't cause all the artists and albums in that genre to rescan.
It's really the proper way to do things, but it required some changes to the data sent between Amarok proper and the collection scanner, so had to be done carefully (as far as I am currently aware I didnt' cause any regressions). Regardless, I'm glad to have it in there and working, and hopefully you will be too.
October 15th, 2009 - 02:04
What about noatime? I think this feature was not present because it caused problem to noatime users.
October 26th, 2009 - 15:43
atime has never been used; only mtime.
October 15th, 2009 - 04:17
Did you check whether mtime changes when renaming/changing a file and not adding/removing it? AFAIK nepomuk does a full rescan of all folders because there is no other way to find all changes that might have happened to files and would require updating the database.
So if the nepomuk devs are right, amarok will not rescan a folder whose content changed and thus have faulty info in the database although the user thought he had enabled “watch folders for changes”. If they are wrong, nepomuk re-scanning the whole “collection” after each kde login is useless.
October 26th, 2009 - 15:47
Renaming a file will cause the mtimes of both the file and directory to change.
What will *not* cause a mtime change is a change in file contents. The Nepomuk devs are right in that without fully rescanning all files you can’t find changes in file contents.
In Amarok, we’ve always ignored this case, going on the general assumption that most people will either rename files at the same time as retagging or retagging them inside Amarok, and we’ve always been forthright about this (and told people to use “touch” when necessary). Yes, it’s flawed, but it’s a tradeoff between speed and functionality (as always).
October 15th, 2009 - 09:08
Cool!
Many thanks!
I’m already anxious to have 2.2.1 in Gentoo!
My collection scans tend to take quite some time, since my collection isn’t perfectly organized and I have some toplevel files and many sorted in a hierarchy – inside that toplevel. It sounds crazy, but it’S the grown result of 7 years of sorting music with different programs and by hand.
October 26th, 2009 - 15:47
Things are going to get even better. Look for a blog post in the next few days.
October 15th, 2009 - 12:17
sweet. How long has this been in trunk? Running nightly (from Neon), and a little while back I had some issues with Amarok crashing during re-scans. I’ll be fairly excited if it doesn’t do that anymore.
Anyways, thanks a lot for your awesome work. It is extremely appreciated, especially by those of us with fairly large music collections. Just out of curiosity, are there other Amarok things that you’ve been interested in working on, or are you going to take a break after this? If the former, well, we always love to read speculative posts about future endeavors
PS. will test the new scanning improvements once I figure out if they’re in the current Neon packages. If only I wasn’t too lazy to build from git…
October 26th, 2009 - 15:48
Well — I haven’t taken a break. In fact, I’ve gone and pretty much taken this speedup work to its logical conclusion. Watch this space…
October 16th, 2009 - 09:43
Lovely news. This takes care of one of my major annoyances with the current release; having Amarok rescan my entire collection (1,368 albums, sorted by Artist – Album/tracks) every time I added an album was getting tiresome on my old Pentium 4.
As for my only other two major annoyances – 1) non-zoomable text in the applicable applets (Wikipedia, Lyrics, etc.) and 2) no visualizer – any hopes of seeing those resolved in the 2.2.x series? These two things seriously cripple Amarok’s functionality on my media center box. :\ I know #2 is dependent on Phonon, but #1 seems (from a non-developer’s standpoint) like it’d be easily implemented.
Keep up the great work!
October 26th, 2009 - 15:49
I don’t know much about #1, but I *believe* that #2 may happen. The necessary bits for visualization just got added to Phonon recently, so now it’s something that can be worked on.
I’d file a bug report for #1.
October 17th, 2009 - 20:39
YAAAAAAAAAAAAAAAAAAAAAAAAAYYYYYYYYYYYYYY!!!!!!!
October 25th, 2009 - 16:56
Nice work. I look forward to trying it out.