Previously, our advice was "touch the folder" or in the worst case "just do a full rescan." Now there's slightly better advice we can give. If you right-click on a folder in the collection setup dialog, and if the folder was already in your collection (i.e. you didn't just check it a moment ago), it will give you the option to rescan that folder.

The text above that field has also been modified slightly to indicate that this is possible. Check it out:

Rescanning a Single Folder

This is in current git master and will be in the 2.2.3 release!

This number was at least three. At the very minimum, assuming the artist, album, genre, composer, and year had been cached, you had a lookup for the uniqueid, a lookup to check for whether the track was part of a compilation, and an insert. This number could be significantly higher in some situations; but these database queries were a large part of the reason that a scan might run, finish disk I/O, and then spend quite a long time eating up CPU time before it was done.

I looked at increasing the number of values I was pulling in using UNION queries, which was currently up to five values (depending on whether they were already cached). I also looked at prepared statements, which it turns out were not likely to give us a large speed bump and are an absolute and utter pain in the ass when using the C API as we are. (We use the C API for historical reasons; also, the official C++ API is not easily available in many distributions, and there are many unofficial contenders.)

Finally, I looked at ripping the guts out of the scan result processor and doing what  several of us devs had decided would probably be the best way to improve speed, but was at the same time the most challenging: replacing the SQL statements with local memory storage, populating these at the beginning of a scan and re-populating the database at the end. It wasn't easy. I selected data structures very carefully for speed, looking at how often each query would be run and the time complexities of insertion and lookup into various types of structures.  In addition, because of a few complex queries using joins, the behavior we needed could only be emulated by using very complex data structures (one of them is a QHash<QPair<int, QString>, QStringList*>; another is a QHash<QString, QLinkedList<QStringList*> *>). I also wanted to minimize memory usage...I did some rough back-of-the-hand calculations which indicated that using my design, for the vast, vast majority of collections, memory usage during the scan was likely to go up by only a couple of megabytes at the very most, which would be reclaimed at the end.

In addition, I implemented logic to drastically decrease the total queries needed for insertion. Since I'm constructing the final queries from the data structures, I could easily do insertion of multiple values at once. So now the database is queried at the start to see the maximum query size it will accept; then values for insertion are appended to the query up until this value is hit, at which point the query is run and a new query begun. By default I think this is one megabyte for most installations, which means that instead of possibly thousands upon thousands of insertion queries (depending on the size of your collection), you'd have to have an absolutely extraordinarily large collection to have more than 25 or so queries for the entire scan. Since there is not only round-trip delay sending data to and receiving responses from the database but also the database must parse each query each time it's run (the purpose of prepared statements is to reduce this), then this makes a drastic difference. To put it another way, the number of queries per track has gone from a minimum of three to an asymptotic value of zero.

Overall, the work seems to have paid off rather nicely. Benchmark reports that I got back indicated anywhere from 30% to 300% gains in the total time for a scan, depending on size of collection and your particular I/O throughput/CPU/etc.

I call this the Ultimate Speed Bump in the title because there is far less that can be done from this point on to increase speed, at least as far as I can currently tell -- this was the Big Mama, the elephant in the room. But hopefully it will increase scan processing enough that further increases won't really be so much of a necessity anymore.

One other data point: the database is now case-sensitive, which is something we've intended to do for a long time but is now being done. It's been a very long-standing feature request that we've finally implemented (we wanted it too), and thankfully it wasn't much work to do this and change the appropriate points in the rest of the code.

(By the by, this will all be in 2.2.1 -- enjoy!)

Professor Bourne is well-known in his field for contributions to open-source bioinformatics software and is a leading advocate of open access to data. Quoting from the UCSD News Center:

Bourne is co-founder of SciVee, the Web 2.0 resource dedicated to the dissemination of scientific research and science-specific research networking. Launched in late 2007 as a collaboration between the National Science Foundation and the San Diego Supercomputer Center (SDSC) at UC San Diego, SciVee has been used by hundreds of thousands of students and professional scientists as a means of learning and sharing their research through online science videos that supplement peer-reviewed journal articles, stimulate discussion, and promote collaboration. SciVee earlier this month announced a number of significant upgrades to its site, along with the addition of 32 new science categories.

Open data access and open source share similar goals, and Professor Bourne's discussion of his experiences with open data access should prove informative and interesting to all.

First up, and the lesser of the two: If a directory is encountered multiple times during scan, the scanner will now only scan them once. There are a few corner cases where this could happen (for instance, if you have a top-level directory specified as well as one of its subdirectories, and the subdirectory's name changes, changing both its and its parent's mtime). Not too useful for most cases, but was implemented while working on...

Second, and the biggie: When doing an incremental recursive scan, Amarok will now no longer scan subdirectories whose mtimes have not changed. This is big news if you have a large, flat collection (which I dislike myself but some really enjoy...to each their own). For instance, in a setup like 3,000 files in your main folder, plus a few thousand files each in a few subfolders for a total of 10,000 tracks, if you added a single file to the top-level folder and had incremental recursive scanning turned on (the default), you'd cause a rescan of your entire music collection -- all 10,000 tracks or so. Now, if you add a single file to the top-level folder, you'll only cause a rescan of 3,000 tracks...which is still a lot, but even more of a reason to use a hierarchy.

This should help scanning time even for those with hierarchies, as adding a new album to an artist won't cause the artist's other album folders to be rescanned, or adding a new artist to a genre won't cause all the artists and albums in that genre to rescan.

It's really the proper way to do things, but it required some changes to the data sent between Amarok proper and the collection scanner, so had to be done carefully (as far as I am currently aware I didnt' cause any regressions). Regardless, I'm glad to have it in there and working, and hopefully you will be too.

I'd heard these whispers recently about "whenever I add or remove an album from the collection and do an incremental update, my collection gets messed up." Okay -- we've heard these whispers for a long time, but there were so many other things that had to be fixed first that it wasn't clear whether this was a symptom of a different problem or a problem of its own...and it could always be fixed with a full rescan.

However, with the collection being much more solid these days and with this being the only super-visible bug left that I knew of, I decided to tackle this. It turns out that, once the rest of the collection was behaving, this wasn't that hard to find...it just took a lot of debug tracing, because it wasn't obvious. Since the cause was the wrong field being used in a DB query to remove some tracks during an incremental scan, with anything other than a very tiny test collection it could quickly start to make no logical sense what was happening.

The reason I say the bug is historical is that the code causing it has been there since May 2008. While admitting that provides some fodder for naysayers and haters to harp on Amarok for such visible and longstanding bugs, I prefer to take the other approach: it means that all the various issues users have found since the total rewrite that was 2.0 *are* being found and *are* being solved. Some of them just take some time; it's a *lot* of code. But the proof is in the 2.2 ChangeLog.

I managed to sneak in the fix a day before 2.2 was tagged, so when Amarok 2.2 comes out scanning (both full and incremental) should be really quite solid. In fact, since this has been fixed in git, I've yet to hear of any more problems, just lots of happy users. It's just another way that the (very close!) 2.2 is going to *rock*.

Then, a little while later, I started getting bug reports of peoples' music disappearing from their collection, and requested some of the reporters send me some files. One of the users did so, and I found something curious in his tags (if I had a penny for every time I've personally seen users have something odd or strange in their tags, I'd have...well, a few dollars at least). Several of his files had full MusicBrainz tags -- with absolutely no data populating them, meaning that the MusicBrainz identifier (and all other MB data) for all of those files was ending up the same (blank) and Amarok was thinking them the same file.

It was a quick fix (use generated non-embedded AFT IDs when the MB tags are empty) but just adds to the evidence that you can never, ever trust users' tags. Also, that users that use your Git-based version or betas really rock for finding this stuff before release...so in case I don't say this enough: thanks users!

Full details are here.

As it turned out, there was a "Kde" network -- but something was odd. To join a work network you have to have an email address affiliated with the network. KDE owns kde.com and kde.org -- so who was this? The only other "KDE" I could find that seemed like it would be legit was the Kentucky Department of Education, and I rather doubted it was them, because they would likely have used all-uppercase KDE as well. So I started an inquiry with Facebook, trying to figure out if either it was someone squatting on our name (and trademark) or whether it was some legit organization -- in which case, would they mind donating the network to us?

After several months of back-and-forth with the people at Facebook, who were very nice (if a bit slow ), I'm happy to say that we've regained the KDE network (properly capitalized) as our own. I still don't know the whole story as to who was there before, and never will due to their privacy policies, but I'll say this:

• If you were in the "Kde" network before and Facebook asked if you would mind donating it to us, and you did, thanks so much!
• If someone was simply squatting in the "Kde" network before, then thanks, Facebook, for kicking them out!

To join the network, go to Settings -> Networks, and enter KDE and your kde.org email address in the appropriate fields.

First, the schema/table changes.

1. We've seen some issues where people have, for whatever reason, ended up with InnoDB tables instead of MyISAM tables. This is probably the result of their DB being created long ago before we were explicitly telling the mysqle startup to skip InnoDB. This mainly causes a problem because some columns cannot be as wide as we'd like them to be when using InnoDB. So, the first thing being done is that an ALTER TABLE is being forced on every table to explicitly convert to MyISAM. In addition, ENGINE parameters are now used during table creation to be more explicit in the future.
2. Some of you might have seen complaints in the debug output about indexes not being able to be created due to a max key length, which by default in MySQL is 1000 (compile-time option). So, some columns have had their widths adjusted so that all indexes are now successfully created.

Now, the other changes:

As we added more features, scanning got slow. Like, really slow. You'd spend more time running SQL queries than actually scanning your files. So I've been aiming to change that.

Over the past week I've committed changes that remove, per track, anywhere from 1 to 6 SQL queries. The exact amount is highly dependent on your file set, but there is a minimum of one less SQL query per track. If you've done a lot of file moves and AFT kicks in, it'll be an even more massive speedup. I'm going to try to do some further tuning, but already results are looking positive.

Nikolaj has reported that his scan time went from 68 seconds to 18 seconds -- more than 3x faster. Mikko didn't notice a speedup, but he said that whereas scanning used to peg his CPU at 100%, it no longer does so. What I want to know is: how does this affect *you*?

If you want to help, do the following:

1. Backup your DB. If you're using external MySQL do a mysqldump, if you're using internal MySQLe backup the mysqle folder in the Amarok data directory.
2. Update to a revision from a week ago...say, 995000.
3. Wipe your DB.
4. Start Amarok -- it will do a full scan because of the empty DB. Time it as it does the scan.
5. Repeat steps 3 & 4, so that you can see what the time is like after caching.
6. Update to current trunk (at least 998470).
7. Repeat step 3.
8. Repeat steps 4 and 5.

Then leave a reply here with your values. If you watch your CPU during each of the scans, report that here too. Thanks!

Please note that this is *not* a post about why KDE is migrating to Git, why this is a good idea/bad idea/neutral idea, etc. This is purely discussing the hosting aspect of Git.

First, I would encourage you to read this kde-scm-interest mail, which I sent to the list on July 2nd. It goes into a good amount of depth as to why Gitorious.org could be beneficial for us, and the rest of this post will assume that you have read that email and the others in the thread, as it will simply update the information therein.

On Monday a large group of interested people, including KDE sysadmins and the guys from Shortcut AS, went to lunch to discuss the technical issues. The output from that discussion is as follows:

• The vast majority of those present feel that Gitorious.org would be the best choice, with the following being the main reasons:
• Shorcut could provide a SLA (Service Level Agreement) guaranteeing a minimum level of service, such as uptime and available bandwidth, providing professional hosting services and easing burden on our system administrators.
• As David Faure noted, user account creation is becoming a large burden on our system administrators, which is not something that we would have to administer if using Gitorious.org.
• It should be noted that the above was not a unanimous opinion.
• KDE does have infrastructure and bandwidth; it could keep one read-only Subversion server available for historical reasons, and convert the rest to serve as backups or possibly load-balancers. Or to put it in a more general fashion, KDE can reduce hosting costs (which will likely be covered by sponsors) by working with Shortcut. It is not a question that this could be done, but rather what the right method would be for doing so.
• The Gitorious developers have a feature branch where they have already fixed one or both of the current showstopping bugs relating to rights within the shared Git repository. They have said that this should be merged into mainline within a week (not sure if they meant a week from then, or from the end of GCDS).
• The hosting could be set up in such a way that it can be accessed via git.kde.org.
• Post-commit-hook functionality will be available; the Shortcut guys are currently working with us to determine how we can migrate or emulate pre-commit-hook functionality.

We have two projects that are chomping at the bit to get onto Git ASAP: Amarok and TagLib. Amarok will be converted first and will serve as the initial guinea pigs to iron out any issues. Barring any major issues being found, TagLib will be converted in short order.

I hope this gives everyone a better idea of KDE's Git-hosting plans. If you haven't checked out Gitorious.org, I encourage you to do so; it's made huge leaps and bounds in the past six months and has become quite a great tool.

Please direct any questions or feedback to the kde-scm-interest mailing list at: kde-scm-interest at kay dee eee dot ooo arr gee, not to the comments section on this blog.

In revision 992942, I finally fixed a bug that has kept AFT working for the playlist in certain situations (although it had previously been working for both saved user playlists and statistics). This means that if you have a track in the playlist, move it to another location, and it is then scanned in that new location (remember, kids, it uses folder mtime to determine whether to scan a folder, so when in doubt do a "touch ."), the track in the playlist should remain valid and play the song in the new location. As the playlist use case was one of the initial reasons for the development of AFT back in Amarok 1.4, you can imagine I'm happy that it's finally (seeming to be) working again in all scenarios, instead of failing in certain situations.

Well...you were wrong.

It may be a bit later than planned -- we wanted to have it in time for 2.1, but it didn't happen -- but as of revision 984572, there is now support for storing an Amarok database on a MySQL server instead of the embedded MySQL database.  There's no configuration dialog in the GUI yet, but it's pretty simple to set up, as explained below.  All you have to do is add a few things into your amarokrc file and make a valid user on the MySQL server instance of your choice -- you don't even need to create the database yourself.  (In fact, you shouldn't -- you should let Amarok create the database so we can ensure that the character set and collation are set right.)

Here's how to do it.

1. Update to at least r984572 (of course, updating to the latest revision is probably your best bet).
2. Wipe your build dir clean and rebuild.  Not necessarily necessary, but as 47 files were changed in that commit, it's not a bad idea.
3. After install, run kbuildsycoca4 --noincremental, just in case. 
4. On your MySQL server, run a command like: "GRANT ALL ON amarokdb.* TO 'amarokuser'@'localhost' IDENTIFIED BY 'mypassword'; FLUSH PRIVILEGES;"  Be sure to substitute for "amarokdb", "amarokuser", "localhost", and "mypassword" as appropriate.
5. Open up your amarokrc file, usually in ~/.kde4/share/config/amarokrc.  Add a [MySQL] section:
   [MySQL]
   UseServer=true
   Database=amarokdb
   Host=localhost
   Password=mypassword
   User=amarokuser
6. Close the file and start Amarok.  It should create the database and start a scan of your files.  If you want to switch back to the embedded collection, simply set "UseServer" to false.

Pretty easy!  Be sure to let me know if you have problems -- file a bug and assign it to "mitchell" at a domain of "kde" plus a dot plus "org".

We've had various bug reports over the years relating to character sets and collation, causing issues with matching searches for music or mis-sorted items.  Well, hopefully no longer.

When you update to 2.2 (recent SVN users, see the note at the end of this post), your Amarok database and tables will be converted to use the 'utf8' character set and 'utf8_unicode_ci' collation as default for any table or column created from this point on.  Every single text/varchar field will also be converted through a two-step process to use 'utf8' as the character set (the data inside was always UTF-8, but there was a possible mismatch between what the data was and what the database thought it was, if your mysql wasn't built to use 'utf8' by default).  In addition, the character set used when talking to the embedded server (the protocol in the socket) will be 'utf8'.

Fixing this mismatch between what the server might have been using for character set/collation and the data we're putting in there should hopefully ensure that sorting and tags work very well for our users with some files wth non-Latin1 tags (probably just about everybody these days).

* Recent SVN users: if your build date is earlier than this post I'd recommend wiping your mysqle directory (not just a full rescan), as the initial commit of the updating code contained a bug that could possibly cause trouble down the line with user playlists...but you bleeding edge users should be expecting database wipes every now and then 

The good news: in current SVN (and thus 2.2) the amarok_afttagger executable will also now handle FLAC and various Ogg-contained formats. Another piece of good news - the amarok_afttagger executable is now contained in the amarok-utilities package, and thus can be run on headless machines without X!  And lastly -- AFT now works with user playlists, so you can move your files around (keep in mind the caveats if you're not using embedded AFT tags) and your playlists will always stay current, in addition to statistics and The Playlist.

The bad news? Something is currently a bit broken somewhere deep inside with Observers which means that The Playlist will only update with the correct new URL once (the metadata observers seem to die after that).  This doesn't seem to be AFT specific but rather seems like a bug that AFT is exposing.  Closing Amarok and reopening it will cause the proper new URLs to be in the playlist.  I'm working on trying to fix that.

(Important note: it writes into the FLAC Xiph comment.  This is the only metadata type actually required by the FLAC spec, and thus is the proper place to put it, but a lot of FLAC files incorrectly only have ID3v2 tags, so depending on the tagger you're using you may only see one or the other.)

Anyone interested in scanning their collection locally instead of across a network connection, or keeping their collection up-to-date when Amarok is closed, should definitely give it a read!

http://amarok.kde.org/wiki/Batch_Mode

This means we have a lot of great work ahead of us.  The first order of business is that everyone that wants to be a mentor needs to sign up at http://socghop.appspot.com.  You'll need a Google account (which does not have to be a Gmail account).  Click the link that says "Apply to become a Mentor" and proceed as instructed.  You should also ensure that you're on the kde-soc-mentor mailing list.

Students: now that we're accepted, get to it with contacting mentors!  The application period opens March 23rd and closes April 3rd, so it's not a huge amount of time to get all the details of your proposals worked out.  Be sure to check the ideas page at http://techbase.kde.org/index.php?title=Projects/Summer_of_Code/2009/Ideas, or contact the appropriate mentor if you have a different idea for a project that you'd like to work on.

I'll be sure to keep everyone informed of events as they occur and deadlines as they arrive.  Here's to a successful summer!

]]> http://jefferai.org/2009/03/gsoc-2009-were-in/feed/ 0 http://jefferai.org/2009/03/gsoc-2009-application-submitted/ http://jefferai.org/2009/03/gsoc-2009-application-submitted/#comments Tue, 10 Mar 2009 21:49:00 +0000 jefferai http://blog.jefferai.org/?p=1050 Today the application for KDE to participate in GSoC 2009 was submitted.  They check out our ideas page when they're evaluating applications, so be sure to get your ideas up there!  We'll hear back by March 18th.  Some other upcoming dates:

• March 18-23: Students discuss applications with mentoring organizations (you can and should do this earlier, of course)
• March 23: Student application period opens
• April 3: Student application period closes

Of course, this hinges on us being accepted...but past years have shown that we have a good shot.

]]> http://jefferai.org/2009/03/gsoc-2009-application-submitted/feed/ 0 http://jefferai.org/2009/03/licensing-to-kill/ http://jefferai.org/2009/03/licensing-to-kill/#comments Wed, 04 Mar 2009 06:43:00 +0000 jefferai http://blog.jefferai.org/?p=1049 Could FLEXlm be one of the world's worst-designed programs?

They've just rechristened it FLEXnet Publisher, and I can only think it's to try to get away from existing FLEXlm stigmas.  It's so bad that according to a VMware engineer I spoke to on the phone a few months ago, in the next release of ESX (whatever the new name is going to be) they're ditching it to go back to their own serial-number based scheme, entirely because of a large amount of hugely negative customer feedback.  This is only one major release after they switched to it.

Let me describe the structure of it (if I were to go into the user interface issues, this would become far too long a post, so maybe another time).  There's a management daemon, called lmgrd, and vendor daemons.  These vendor daemons are (often along with the management daemon) provided to you, often without an installer and simply as a bunch of files.  There's also quite often a node-locking generator (based on things like hostname, MAC address, and all manner of things easily faked in a VM) provided to you by the vendors, which as far as I can tell is different for every vendor.  This is possibly so that they can build in their own criteria, but more likely because the FLEXlm people are lazy and can't be bothered to provide a comprehensive set of criteria on their own.

Now, why is there a vendor daemon and a management daemon?  Good question, and I don't know the authoritative answer, but from using it I can make an educated guess or two: either to allow vendors to validate some part of the licenses in their own manner, or to simply confuse and annoy the hell out of you.  See, the different vendor daemons are built against some version of the FLEXlm software, and they may, or may not, have been built against the same version.  This may, or may not, cause problems, including crashes.  It may explain why every time the license server reboots (thanks, generally, to automatic Windows updates), lmgrd fails to start up successfully.

Now, you can run multiple instances of lmgrd -- if you can figure out how to get it installed so that it starts up on bootup, since there isn't much in the way of help in the official help PDF, and only one of the three vendors whose licenses I deal with actually provided an installer (thanks, VMware, I'll be using your installer long after you've switched off of FLEXlm).  This lets you have each license under a different copy of lmgrd, perhaps so that you can attempt matching versions, but mainly so that if one of the lmgrd instances crashes, it will only take down that license.  (Actually, it's because in this configuration you need to run them on multiple machines, so if one machine goes down, most of your licenses stay up.  But I know what they really meant.)

Alternately, you can combine licenses into a single file, a process that the manual warns is time-consuming and error-prone.

Finally, you can simply have all your licenses be handled by a single lmgrd instance.  Now, this isn't really a bad idea, except for port numbers.

See, each vendor daemon needs to run on its own TCP port number.  You can specify the port number in the license file, except when you can't because some vendor hardcoded it into their software.  If you don't specify a port number in the license file, the vendor daemon will be given a port number starting at 27000 on up, giving you an address like 270...@mylicenses.thissucks.com.  Now, because everyone likes autoconfiguration, if in your program you leave out the port number (giving you something like @mylicenses.thissucks.com), then it will automatically search ports 27000 through 27009 for a matching vendor daemon.  Unfortunately, I haven't seen a single vendor product that doesn't puke on such an address, claiming it is invalid because of faulty validation, even when the FLEXlm manual in front of me says it's perfectly legal.

If you don't then modify your license files to specify ports (which you have to remember to do every time you get a new license file), then you better be aware of when the license server reboots, because the ports are given out in the order that the services come up.  So if you're in my situation, where the first of the vendor daemons always crashes the first time it tries to load on bootup, then the other two will have their normal ports decreased by 1.

I haven't even gotten to things like functions to reread license files that don't actually reread them, a UI that doesn't tell you if a server is running or stopped until you hit the buttons to try to run it or stop it, a status window that displays a large amount of license data in a tiny, non-resizable window, and more.  The product feels like it was developed in 1988 (it was) and is a study in market leader stagnation.  I would be very surprised if they have a single developer left and haven't fired everyone to just sit there and watch the money roll in.

If license servers are necessary for some product you're creating, there are other license servers out there that simply run on a well-known port and don't require any of this idiocy.  Unfortunately, in my experience I've seen more companies switch to FLEXlm than away from it. I hope this isn't an industry-wide trend.

Where's the KDE angle?  There isn't one specifically, other than this: if someone in the KDE community thinks this is a good design, or can see themselves designing such a scheme, please just leave.  Now.  (Don't worry, I'm sure this doesn't apply to any of you. )

]]> http://jefferai.org/2009/03/licensing-to-kill/feed/ 11 http://jefferai.org/2009/02/amarok-power-user-feature-batch-mode-collection-scanning/ http://jefferai.org/2009/02/amarok-power-user-feature-batch-mode-collection-scanning/#comments Fri, 27 Feb 2009 23:58:00 +0000 jefferai http://blog.jefferai.org/?p=1048 A long-requested feature has been a way to decouple Amarok's collection scanning from its GUI.  There are various use-cases for this.  For one, it can actually help us with debugging, by allowing us to control the inputs into the scan parser.  For another, many people have all of their music stored on a single machine, and would like to do the scanning locally where it's fast instead of on their e.g. laptop running Amarok, where it's over wireless and slow.

Yesterday and today (as of r933010) I put half of the solution into trunk.  I say half, because full collection rescans are now supported in batch modes, but I am still working on the methodology for incremental scans (I have a few ideas, but have to sort out which is the most reasonable/doable/makes the most sense). Below, I'll explain how to do it.  Keep in mind that this is designed to be (lightly) scripted, not done by hand...so it can be done by hand (which I did during testing) but it has some safeguards in place so that if you script it, and forget about it, Amarok still works normally.  However, there are actually some interesting things you can do now if you script the scanner...

One important thing to note: the scanner requires various bits of Amarok code, mainly centering around the extra taglib plugins we support.  So you're unlikely to have it work on a machine that doesn't have A2 installed, and certainly won't be able to compile it without the rest of the Amarok source, although you may be able to get it to work in a binary-only fashion with just kdelibs and taglib...YMMV.

So, here's the flow:

1. Run amarokcollectionscanner with the -b or --batch option.  Although this doesn't directly modify the output, it sets some internal flags (some for safety, some to enable other options).  The output goes to stdout; save this output as amarokcollectionscanner_batchfullscan.xml (yes, it must be that specific filename).  You'll probably want to use the -r flag too...check --help for more options.
2. Cool feature: if your scan is of files that have a different mount point on the local machine than the machine that will be using the output, add the --rpath option and give it the mountpoint of the directory on the consuming machine.  For instance, if on the local machine your music is at /opt/music, and this directory (music) is mounted as /mnt/music on the remote machine, run amarokcollectionscanner from /opt and use --batch --rpath /mnt/music.
3. Cool feature: You can scan multiple directories and simply append the output to the end of the file from step #1.  Yes, directly append, including the xml version/encoding header.  Amarok's scan manager will detect and handle this.  You can even scan directories that are not defined as collection directories in Amarok (these entries will work fine, but the next time you do a full rescan from within Amarok without using the saved output, they will be gone).
4. Place the amarokcollectionscanner_batchfullscan.xml file in your Amarok data dir.  Usually, this is ~/.kde4/share/apps/amarok.
5. Trigger a full rescan from inside Amarok (in Settings->Collection).  It should be super-fast.    Note that this step deletes the amarokcollectionscanner_batchfullscan.xml file.  This is on purpose, so if you want to keep re-using it, you'll have to keep copying it over.

As you can see, it's designed for scripting, but it's not terribly complicated even to do by hand.  I'll post more as I get incremental scanning working, and will at some point put a Wiki page up that has all of these instructions in a less transient form.

]]> http://jefferai.org/2009/02/amarok-power-user-feature-batch-mode-collection-scanning/feed/ 3 http://jefferai.org/2009/02/camp-kde-videos-come-and-get-em/ http://jefferai.org/2009/02/camp-kde-videos-come-and-get-em/#comments Wed, 25 Feb 2009 18:41:00 +0000 jefferai http://blog.jefferai.org/?p=1047 After a month's delay, they're done.

In the interim (after the delays mentioned in my last post on this topic), my poor, underpowered desktop machine has endured transcode after transcode (from the original source material) and my Internet connection upload after upload as I tried to figure out just why Blip.tv wouldn't work with X or Y.  (In fact, I can quantify these: X is Vorbis/Theora, which produced awful audio and very desynchronized audio/video upon their conversion to .flv; Y is a lot of things related to the original anamorphic encoding of the videos, which Blip.tv can't handle, and finding the right combination of settings and flags and adjustements to make the aspect ratios come out so that everyone didn't look like Gumby®.)

This was followed by a few days of uploading; Blip seems to max out uploading speeds somewhere between 100kbit and 200kbit, so uploading almost 6GB of data, one chunk at a time, took a bit.

The end result is thirteen videos, each Xvid-encoded (at least it's OSS, although patent-encumbered...see Vorbis/Theora problems above) with a Matroska container.  You can get to them through my show or to individual videos directly (these might not be in strict as-presented order):

• Welcome!
• Diversity in KDE
• Akonadi
• KDE and Business Software
• KDE4 and MS Windows
• libplasma in Applications
• Accelerating Graphics
• CMake / CTest / CPack
• Taming the Leopard
• Bringing the Free Desktop to the Mobile World
• KDE & Distros
• KDE-Games, KDE-Edu, and Avogadro
• A Case for Open-Source Coders in the Enterprise

Enjoy!

The title says it all.  They're available from my PPA.  Enjoy!

The Bad News

I've run into a series of setbacks regarding getting them prepared for posting.  The first was that the Monday I got back from Jamaica, I had the following schedule at work:

• 9:15 AM: Check email
• 9:16 AM: "My details have been confirmed for my upcoming trip?  What trip?"
• 2:00 PM: At the airport awaiting departure, thinking that I *knew* I should have checked work email sometime over the weekend...

Regardless, I did manage to have some of the files with me, and did actually get a few of the videos done during that week that I was away.  I then got held back a bit waiting for the videos on Wade's camera to arrive -- they were 16GB or so in total, and so he had to fetch them off the tapes, encode them, and get them uploaded to me.  After that, I started to put Wade's videos together with slides, only to find out that the videos would not properly work inside kdenlive.  Or so I thought.  After a period of long transcodings that I performed during my free time over the last few days, I find out this morning that the issue with the videos was a massive dose of PEBKAC.

There's only one other piece of bad news, which is that due to a bug in (ffmpeg? libmlt?) kdenlive's Ogg/Theora export is borked.  So, I've ended up coding in XviD, which while using a patented algorithm is at least otherwise free software.  I've thought about transcoding everything to Ogg/Theora afterwards, and I may end up doing so before posting them up, if the quality is okay.  And before you ask -- it didn't seem right to post them up without having Wade's keynote up there to kick things off, which is why I've not posted some of the ones that are already done up before now.

The Good News

The good news is that 11/13 of the videos are done, and I forsee no obstacles in getting the rest of them to behave.  So I am hopeful that I can start getting them up to Blip.tv within a couple days.  Thanks for bearing with me.

Anyways, one of the reasons I'm using it is to splice the slides into the videos, because they're just not readable inside the videos for the most part. So I needed a way to turn each slide into some sort of an image.

It turns out that OO.o doesn't have this capability natively, but some users on the OO.o Forum came up with a script at the bottom of this page to export each slide into PDF. I modified it to do the following:

• Export to JPG instead of PDF (PNG export didn't work)
• Add extra 0s to numbers such that you always have three digits

The code is pasted below. One really fun (not) thing I found out: VBA (or just OO.o's implementation of it) doesn't really do type checking. As a result, if r is a string instead of an integer (which I had forgotten), the following code will always execute as True:

If r < 10 Then

Anyways, here is the code, in case it helps anyone at some point:

REM  *****  BASIC  *****

    End If
  Next
End Sub

The latest perpetrator of bile and venom is one Antal István Miklós. This blathering idiot Web Developer called me out personally on his Great Blog (yes, that's really what it's called, if you want to see his post), and turned my well-received talk at the KDE 4.0 Release Event into complaint-a-thon.

I'm not going to pick apart all of the technical wrongheadedness he portrays with MySQL/e, Akonadi, and the capabilities therein -- Nikolaj posted a nice response to the blog, and the guy would be very well served to *fully* read my posting about mysqle (as well as many of the comments). Much of the blog can also easily be decoded as baseless, factless trolling ("amarok1.x is the slowest KDE3 program, if not it's surely in the top 3 slowest KDE3 programs"), and the-developers-don't-agree-with-me-so-they're-wrong syndrome. But I *am* going to defend my statements at the Release Event.

Let me explain, up front, the format of my Release Event talk. I presented a short introduction to Amarok, for those that did not know it. I then stated some drawbacks we found with the KDE3 platform. With these drawbacks, plus a desire to go cross-platform like many other media players (VNC, Banshee, iTunes, etc.), we had considered the possibility of switching to a Qt-only architecture. But then, KDE4 comes along, with elegant solutions to our problems -- Phonon, Solid, Plasma, targeting multiple platforms, and more. Boom -- the thoughts of Qt-only go out of our heads, and we commit ourselves fully to KDE4.

Far from a long series of complaints, it's a success story, showing how the benefits the KDE4 platform offer to us solved our problems, and how they could solve yours too. Apparently, however, it simply shows -- from Antal's blog post title, and probably because he hasn't bothered to watch past five minutes in -- how we just don't get KDE4.

This may not be apparent to everyone, but Amarok was an early poster child for adoption of many of the Pillars of KDE. We are the only application, to date, that has embedded Plasma inside of our application (with our developers doing a large amount of work to make that possible). (Update: we are technically the first, outside of the plasma workspace, but there are others playing with that now.) Device detection completely relies on Solid (which is one reason Mac and Windows ports have no device support right now). And we have completely standardized on Phonon for our media engine. We've also had Oxygen team members working on our icons and our interface.

It's hard to imagine ways for us to more fully integrate with KDE4 than what we are doing. We've gone for KDE4 whole-hog, and it's ludicrous to suggest otherwise. Picking out random Pillars that we don't fully integrate with (yet) does not mean that we are not KDE4-oriented. After all, right now we don't have a use for Marble (who knows? that could change) -- but does that mean we don't get KDE4?

it just reminds me of one of the KDE4.0 release event, where a KDE dev complained that how KDE3 sucked, because they couldn't port Amarok to Windows, and KHTML had bad performance

It'd be nice for Antal to realize that there is a difference between complaining, and listing drawbacks of a platform. (This doesn't really fit into how trolls work, of course.) Yes, I said that two of KDE3's drawbacks were that it made it impossible to port Amarok to Windows (and Mac) and that KHTML rendering was found to be slow. No, I did not say that KDE3 sucked.

But there's nothing new here. It's not like Amarok was alone in wanting to port to other platforms. The Release Event had showcases of KDE4 applications running on both Windows and Mac. But Antal has this fixation that Windows and Mac are suddenly all we care about, taking an out-of-context "consider the majority" statement someone (he doesn't say who) made on IRC about some topic (he doesn't say what, only that it's vaguely somehow about performance):

Using mysqle mostly benefits non-KDE4 desktops, because as I said earlier KDE4 will probably have a mysql server anyway, but isn't improving the KDE4 user experiance top release priority anymore? Is amarok on Windows on Mac more important than getting the best out of amarok on KDE4?

[then, later]

What did the people in the IRC channel had to say about this?

.......

My favorite quote from here is: "consider the majority"

It's like saying: "consider the majority, which are Windows and Mac users, and screw the KDE4 users"

I think Antal fails to realize that KDE is not just a desktop. Windows and Mac users that might be using Amarok are going to be using *lots* of KDE technologies in the process. Regardless of his mistake, there is certainly no evidence that Amarok cares more about Windows and Mac users, or thinks them the majority of our users. Speaking as a developer, I can tell you that the exact opposite is true.

Antal also clearly doesn't realize that Akonadi is not a requirement of KDE to run (even if it's installed), and therefore the best Amarok could do would be to integrate with Akonadi, but not to depend or rely upon it. Maybe he kind of gets it when he says, my emphasis, "KDE4 will probably have a mysql server anyway" -- we can't rely on probably, or maybe. We need to use what works, always.

He even contradicts himself:

He begins with complaining, how slow was rendering amarok's context with KHTML, so it looks like performance matters in amarok, not that anyone forced them to use KHTML for rendering context...

Make up your mind, Antal. You're right that no one forced us to use KHTML for rendering context, although WebKit wasn't available back then, and hooking into Mozilla was a non-starter. But you want us to be integrating with other KDE technologies...right?

One last point:

Jeff Mitchell the developer who spoke at the event that I was referring to, referenced KDE as a family, but where is the love now? The lack of communication between Amarok and the rest of KDE4(Akonandi) doens't seem to back up Amarok as being a family member.

It is not surprising, given how little he understands of Amarok, KDE, and the integration thereof, that he thinks both that there is a lack of communication between Amarok and the rest of KDE4, and that he implies that Akonadi is the entire rest of KDE4.

I've covered a small fraction of the untruths and inaccuracies in his post, but it's enough -- I've made my points. I love KDE, I have not publicly disparaged it, we Amarok developers are fully committed to the platform, and we are not putting the Windows and Mac ports at a higher priority than the *nix base.

Any comments will be read, but I may decide not to post them.

I want to point out first that I said that MySQL is going to be Amarok's only SQL backend. A2's collection system is very powerful. Just take a look at how varied music sources from Shoutcast, Jamendo, Magnatune, Ampache, MP3Tunes, as well as local sources like iPods and your local file system, are treated as equals in A2. A collection is a collection, and is limited only by what capabilities it advertises it can support (and of course, it can supply its own custom capabilities). It's not currently enabled, I don't think, but there's a Nepomuk-based collection option too. So take heart -- this change only affects Amarok's internal SQL collection, and not other sources (although those sources can store information in the SQL database if they wish to cache information).

Since I mentioned Nepomuk, it's time to discuss another common question/demand/complaint: KDE has this nice Strigi-Nepomuk thing going on...why aren't we using it for scanning music and storing information? There are a couple main reasons. The first is that Strigi and Nepomuk are optional, not required. (Update: Strigi is required, but Soprano isn't, so Nepomuk as a whole is still optional.) We can't rely on the user installing them, and even if they are installed, we can't rely on the user to configure them properly (remember that we're going cross-platform, making it even less likely). The second reason is speed: Amarok's custom collection scanner is extremely fast and pulls out specific pieces of information with TagLib. Strigi is, by comparison, very slow (it calculates hashes of all files, which means it needs to read the entire file) and pulls out less information. (Update: According to the Strigi developer, and despite what is said on kde-apps.org, Wikipedia, and even the author's own home page, it does not calculate hashes by default. So it's possible that Strigi, if properly configured, could be as fast as Amarok's internal scanner, although whether it would pull out all necessary information, I don't know. If it's configured to calculate SHA1 hashes of all files, then it will indeed be far slower.) On a local hard drive, it may not be a big issue, but it sure is a huge issue when you throw networked storage into the picture, which is a very common scenario. I've also heard, though don't remember specifics, that querying and such through Nepomuk is rather slow, compared to a normal SQL database.  Regardless, though, remember that when the Nepomuk-based collection is finished, tracks sourced through a Nepomuk-based collection will have their metadata changes saved back to Nepomuk. So, it's not that the SQL collection is in place of Nepomuk -- they are entirely independent. (Update: I forgot to mention that a Nepomuk collection already exists. It was developed by a GSoCer over the summer. I'm not sure what its status is as far as making the 2.0 release, but we Amarokers both like Strigi/Nepomuk and are excited about the idea of opening up the app and having all your music available right then and there with no pre-configuration. But there is a place for the SQL collection too. As I said: they are complimentary technologies.)

With those topics out of the way, on to the meat.

First, it is important to understand an important pair of facts. Number one: we are not database guys. Sure, we can store data in them, and more or less come up with a working schema, but none of us are gurus/wizards/jedis/etc. This leads in to number two: maintaining three databases was driving us crazy. Every time a minor schema change was needed, it had to be coded up for all three types of databases. Modifying a schema could be trivial for one database type, and super difficult (or impossible) for another. People would report bugs that we couldn't reproduce, only to find out that it was because we didn't quite understand how one database or another behaved (or in some cases, none of the active devs were using that type). And so on. So from the beginning of A2 development (and in our fantasies during A1 development) we knew we wanted just one database.

(We did actually look at abstraction layers like QtSQL and others. I'm not going to comment on them much, as I didn't do the evaluation, but in general they were found to not be flexible enough to handle all of our needs without doing some custom SQL coding (especially in the cases of things like schema changes), which kind of defeats the point. If you want to know more/want to insist that they are, try asking eean, as I think he did the evaluations.)

Now we had to choose the type. At first, SQLite seemed like a good choice. Using transactions, it's decently fast. It's pretty stable (those that complain about odd MySQL bugs should talk to markey, as he, being the SQLite maintainer in 1.4, can attest that SQLite's had its fair share). However, there were a few problems that in the end knocked it out of the running. The first problem is performance. Although for people with small collections it performs fairly well, people with large collections that switched to the MySQL or PostgreSQL backends in A1 would report enormous speed gains when operations performing complex or many queries were performed, such as adding many entries to the playlist, scanning files, or filtering/searching in the collection. Since we want to accommodate users with large collections just as well as those with smaller collections, and since digital music collections aren't getting smaller, the speed increase for our users with large collections was quite important. Many of our developers, after the switch to mysqle (as we call it, though that's not the official name), have noticed huge speed increases in their day-to-day use of A2, so that speed increase is carrying through to the embedded server as well as the normal server. That was the first knock against SQLite.

The other blow for SQLite came for a totally different reason. Many users (myself included) have multiple computers sharing a single Amarok database. Assuming all the computers have access to the music at the same mount point (and a few other things are configured right), this allows you to scan once, play everywhere, update the same ratings no matter where you play it, and more. Even if your aren't sharing the database among multiple computers, many users want their database stored on a particular server for speed, security, or backup reasons. If you think either of these isn't a common use-case, you'd be quite wrong. MySQL and PostrgreSQL were quite happy with this workload. It's a total no-go for SQLite, simply because it's designed for a different purpose. So SQLite had two big knocks against it. K.O.

However, just as we can't rely on the user to set up Strigi/Nepomuk correctly, we can't rely on them to get their tables set up in MySQL or PostgreSQL. So we needed the database to be embeddable, so that it could just work for the user without any setup necessary on their part. MySQL, with libmysqld, had the seeds of this in the 4.1 series, it works decently in 5.0, and it's becoming fully supported (AFAIK) in 5.1. PostgreSQL, on the other hand, does not have any such thing. (They have an interesting and cool concept of their own of embedded SQL though. Update: apparently that is part of the SQL standard. Still pretty cool. Still totally different from what we mean when we are talking about an embedded server.)

So this leaves us with -- as you guessed -- MySQL. It may not be any particular person's favorite database (although it is for plenty), and I don't know how much overhead it *really* has in embedded form, but it fit the bill. It's both embeddable and can run standalone on the local or a separate machine (yes, this is not supported yet in A2, but it will be). It is fast and robust for large collections. It is well understood by the development team. And most of all, it is a single-backend solution that fills all of our needs.

If you're still unhappy about our decision, I'm sorry. We try to please most and can't please everyone. But we're the ones that develop and support this thing, and so we made a decision based both upon our needs as developers and the real-world use-cases from the collective feedback of thousands of users that have contacted us over the last few years. Please remember that even if most of the comments on the Dot, or to this post, (i.e. much of the sudden visible feedback) are from people that are unhappy with our decision, it is a decision that will actually suit the vast, vast majority of our users better than the other options we currently have.

We're a project that is known for being good to our users -- we listen to them, we try to implement features they want, try to be responsive with support. It's one of the things that got us where we are today. So please, dear readers -- put some faith in us. This has not been an easy decision -- we've discussed, we've argued, we've thrown things, we've made up, we've had an after-the-make-up orgy or two -- but in the end it's what we collectively felt was the right way to go, and we feel that, in the long run, it will make Amarok even mores awesomer. Hopefully you'll feel that way too.

"We can rebuild him. We have the technology. Better than before. Better, stronger, faster."

A little-known feature in Amarok 1, starting at about 1.4.3, was what was known as Amarok File Tracking, or AFT.  For every single file in your collection, on scan, a unique identifier (UID) was generated from some of the file's attributes.  If you moved your tracks around your folders, as the incremental scan kicked in, the UID would allow for the file to be identified, and integration throughout Amarok would mean that your statistics, your cached lyrics, and the current playlist would all be updated with the new path.  No longer did you have to worry that moving around your files would mean losing years of statistics.  Or losing your files.

But I'm getting ahead of myself.

See, AFT wasn't born AFT.  AFT could not track both a file metadata change and a file location change at once, because the UID was being based on file properties such as file length, plus a portion of the file itself hashed together. So you could still lose track of your files.  This was a limitation that was known in advance.

It was also a limitation that didn't originally exist.  As I said, AFT wasn't born AFT.  It was born as Advanced Tag Features, or ATF.  ATF was the same idea, but a little different -- it would store the generated UID directly in the file's metadata.  This allowed for superb file tracking capabilities, because unlike generating a UID from a part of a file, if that part of the file changed, you'd still have your UID.  In fact, the only way you *couldn't* track your file was if you either removed the file's tag entirely (or some other program removed the UID when it shouldn't), or if you removed the corresponding information from Amarok's database. (There are some downsides to this scheme: only certain file types are supported, for instance, determined by the kind of tag they use and the tag's ability to store this kind of information.)

So why the change?  Well, ATF had a problem, which was related to the structure of Amarok itself, and Amarok's historical penchant for crashing (which got much better as the 1.4 series progressed).  The outcome is possibly worthy of an entry in The Daily WTF.  In gory detail, here's the problem.

   1. Amarok would start a collection scan.  The collection scanner was the entity responsible for adding the UIDs to the file metadata.  Important note: the collection scan was a separate process.
   2. Amarok would crash, leaving the collection scan running, although not communicating with anything.  This scanner could be very slow if it was adding the UIDs, depending on whether padding had to be added to the file's tag.  If this was the case, the entire file would have to be rewritten.
   3. Amarok would be restarted by the user.  Another collection scan process would start.  Becuase UIDs would already exist for the early files, it would very quickly catch up to the first collection scan process.
   4. You now had two collection scan processes generating and writing UIDs at the same time to the same file.  If you were lucky, this would mess up your tag.  If you were unlucky, this toasted your entire file.
   5. Repeat step 4 for the rest of the scan.

ATF was never released in this state, but it did get turned on in SVN.  And a few unlucky users had far too many files end up corrupted, depending on how crashy things became for them.  After we finally realized what the issue was, a user came forward on the mailing list (still trying to find the exact mail or user) proposing a solution that I believe they'd seen in a class.  Essentially, the solution relies on modifications to temporary, uniquely named files instead of the original file, using MD5 checksums to find out of the original file has changed while writing the new file, then using filesystem atomicity guarantees to move the new file back over the old one.  This became the MetaBundleSaver, and it worked quite well, but it was also extremely slow compared to a normal scan.  And most importantly, no one was quite trusting of the whole ATF scheme any more.

So, ATF was renamed to AFT and with it came a new algorithm that wouldn't touch anyone's files, but couldn't track as well.

A couple weeks ago, I added AFT to Amarok 2's SqlCollection.  Enjoy, everyone -- statistics, lyrics, and the playlist are already supported, with support for stored playlists coming eventually.  But there's more.

Fast forward to today (okay, two days ago).  I'm taking a shower -- Wade does insist that there's something about showers and KDE coders -- and I had a thought, which was essentially: there's absolutely no reason why Amarok 2 can't use a UID inside a file, if one exists, for superior tracking, and if not, generate a read-only type for normal tracking.

So I created a utility that is built and installed with Amarok 2.  It's called amarok_afttagger, and it will write UIDs into your files, using a class ported from MetaBundleSaver and called SafeFileSaver to ensure that files are not overwritten/interleaved, even if you run the process twice or three times at once.  It optionally supports recursion if you want to pass in directories, and it can also remove UIDs from your files if you like.  Right now it supports MP3s only, but Vorbis and FLAC support will be coming soon.

I've tested it extensively.  I've added UIDs to files, removed them from files, regenerated the ones in files, over and over, and still everything is cherry.  And Amarok 2, when it finds these files, can do some awesomely robust file tracking.

I encourage people to give it a run on their MP3s and check it out -- if you're worried by all the Dark Ages info up above and don't have faith in the implemented solution, back up your files first, or operate on a copy of them, until you're satisfied it won't harm your files.  And if you still don't want to do it, you can enjoy the less awesome but still awesome power of the non-embedded UID file tracking.

Now, I promised this would talk about a possible KDE library.  I'll eventually be submitting the SafeFileSaver class for hopeful inclusion into kdelibs, so that any application that is worried about data integrity and needs to write to a user's files can take advantage of it.  It's very simple to use -- you simply give it a file path, and then operate on the file path that's returned to you when you call prepareToSave(), instead of the original one.  When you're all done, you call doSave() and it will perform the necessary functions.  That's it.

Hope this has been enjoyable, and enjoy AFT in Amarok 2.  Play with it and be amazed.  Use amarok_afttagger on your files and be even more amazed.  More information is available here: http://amarok.kde.org/wiki/AFT

So iPod support should be pretty well set by this next weekend, and the next thing on my list is support for MTP devices.

Now, on to the subject line -- the PUD now contains an average of 400% more droppiness. That's because you can now drop on the text of an item instead of just the icon, which means a far larger drop target (and apparently this is how people expected it to behave in the first place).

Next on the agenda is to have text change color on hover so you're sure of what you're dropping onto. Complete with a nice looking fade, of course...

]]> http://jefferai.org/2008/06/popup-dropper-now-with-400-more-droppinness/feed/ 5 http://jefferai.org/2008/06/introducing-the-popup-dropper/ http://jefferai.org/2008/06/introducing-the-popup-dropper/#comments Wed, 18 Jun 2008 23:15:20 +0000 jefferai http://blog.jefferai.org/?p=1035 About a year ago we had an issue in Amarok 2 development because we wanted to be able to move portable device track access into collections (this is happening with Alejandro's GSoC project), but we couldn't figure out a good paradigm for accessing the various device specific functions. There were some other concerns too, about the right-click menus getting too overloaded, making it hard to find common functions. Finally, when people wanted to drag music to the playlist, they now had to cross the entire window, whereas before it was a quick drag and drop...a lot less mouse movement, and possibly the difference between picking the mouse up and not.

So I had a brainstorm. We have this ginormous context view area in the middle of A2. What if we could put it to good use, and allow it to act as a surrogate menu? For many people, dragging the mouse a bit to drop items on a big target is much easier than navigating small right-click menus, where it's easy to overshoot your destination and accidentally close submenus. Thus the PUD -- the PopUp Dropper -- was born. My first sketch of it is here:

(Here's the preceding page that talks about collection groups. If you like that idea, be sure to let us know at ama...@kde.org.)

The PUD actually lived for a long time in the Amarok source tree, but there were problems. One was that it tied in too tightly to the existing Amarok context view code -- something that, as that code was refactored over and over again, was painfully obvious would not work. The other was that the actual popping up and drawing was painfully slow. It seems drawing a QGraphicsView on top of another one wasn't making Qt happy. Fortunately, as of Qt 4.4, this problem has been solved.

The final nail in the coffin (related to the first one actually) is that some of the people that saw the work wanted to see it available for all of KDE to use. This approach wasn't going to work.

When Qt 4.4 hit qt-copy, I started to play around with things again. Soon enough I was addicted to making it work. I took the Drag 'n Drop Robot example and started modifying it to use the PUD library I was making. The goals: make it Qt-only, make it very customizable so it could work in any situation, and make it actually able to replace a right-click menu.

I'm happy to say that those goals have been met. Here's a list of current features:

• Runs on top of any QWidget (theoretically, though not fully tested).
• Two ways of creating submenus: either call addOverlay(), which essentially gives you a "clean slate" that you start afresh, and that runs on top of the current menu when show() is called, or you create a second PUD and pass it into addSubmenu() on the first, which is quite similar to the way a QMenu would be constructed.
• Can pop or fade in/out with configurable fade timing on a per-submenu basis.
• Background color and transparency, as well as text, is configurable on a per-submenu basis.
• When the mouse leaves the PUD area, it disappears (or the top layer disappears, and keeping it outside makes subsequent layers disappear)...configurable.
• The PUD items contain a PUD action, allowing for both drops (which calls triggered()) and hovers (which calls hovered())...each one individually configurable, of course.
• The items are represented by a SVG element you pass in, along with a renderer (or the default is to use the PUD's renderer).
• The items arrange themselves automatically (submenu offset is configurable).
• Many more things. All of them configurable (sense a trend?).

I really wanted to make a video available, but I could get neither xvidcap, nor vnc2swf, nor captury working at all. recordmydesktop worked, but interfered with the drops, so it wouldn't work as intended. Instead, I'm making an "alpha" version of the PUD available for people to see (I had been using a local git repository, which is included in the tarball). Don't worry...despite me calling it alpha, it's quite stable (in fact, as you'll see, it's already used in Amarok). No dependencies are needed, just qmake the dragdroprobot.pro file, make, and run the dragdroprobot file. Caveats: it's not commented (see coloritem.cpp for usage examples), I know of a few bugs, it's not finished (I have several more things to add to it), there is at least one crash that I can't reproduce reliably and have not found, there are some things I need to test but have so far been unable to (such as more than one submenu layer), and note the copyrights and distribution requirements on the files -- it's a modified example from Qt. Preferably run in a Qt 4.4 environment (i.e. in a KDE trunk login) to get all the benefits of Alien, or else it might be rather slow depending on your system. But with that all said, let me know how you like it.

Finally, it's been updated in the Amarok source tree. Now that I got the submenu support working, Nikolaj has gotten it integrated. In recent builds of Amarok you can simply start dragging an item from your collection and see it in motion. Some screenshots follow. The first is showing the normal menu, the second shows the submenu that pops up when you hover over the Organize Files item (you can see the previous menu behind it, through the transparency).

So what's next? Hopefully, this will eventually make it in some way shape or form into kdelibs, assuming the API stays stable (it was written in the kdelibs style, with a d pointer and private classes), making it easy for KDE apps to take advantage of it without being an external dependency. And as it gets completed I'll be making it available separately for any Qt app out there that wants to use it (not sure what license it will be under...either GPL or BSD). For now, if you want to see the latest versions, keep your eye on the Amarok source code in the src/popupdropper directory (it has a few modifications to my "master" source simply to make it build in the Amarok source tree...specifically, adding moc file includes since it's not using qmake, and exporting PopupDropperAction).

Comments appreciated!

]]> http://jefferai.org/2008/06/introducing-the-popup-dropper/feed/ 20 http://jefferai.org/2008/04/amarok-soc-media-devices-awesome-ipod-support/ http://jefferai.org/2008/04/amarok-soc-media-devices-awesome-ipod-support/#comments Mon, 28 Apr 2008 14:52:54 +0000 jefferai http://blog.jefferai.org/?p=1034 Introducing Alejandro Wainzinger (xevix on IRC), who is going to be working on media device support in Amarok for SoC 2008:

My name is Alejandro Wainzinger, and I'm going for a Computer Science B.S. at the University of California Santa Cruz, USA. This summer, I'll be bringing back media device support to Amarok for Apple iPods, MTP and generic devices, and making them fly. I chose this project because I own an iPod, and got frustrated with the speed of loading an iPod with a large database, sync'ing of songs/playlists and album art, unlogged crashes after trying to put a 10,000 song queue onto the iPod, and slightly unreliable iPod model detection. That said, I loved having iPod capability in Amarok, and I couldn't see Amarok 2 without media device support.

Alejandro is going to work on getting normal functionality for all three types of devices, and then really taking the iPod plugin to town.

Rumor has it he's then going to attempt making collections out of the devices...

Another reason for the long delay was a partially failed experiment that aimed to provide support for many different media devices via a centralized kioslave. It's not that this couldn't still happen, at some point, but it becomes extremely difficult to map protocols that have no notion of filesystems into a filesystem, and have filesystem clients behave properly. There was another foray into a different system that also ended badly -- mainly because there was no way I'd get it finished within a year, much less within a few months.

In the end it was decided that Amarok's device plugin system is really pretty decent, providing relatively bug-free and easy device management for almost every device on the market...and if it ain't broke, don't fix it. Of course, it's using tons of deprecated classes and methods, but as Trolltech says (my emphasis), "we recommend against using these classes in new code." Good point. So let's get stuff working, and then we can try to design a new, better system in parallel -- but at least something will work in the meantime.

So with that decision made, I worked on integrating with Solid. I'm happy to report that today, I plugged in my Creative Zen MicroPhoto, and it was instantly detected and the correct plugin selected. All I had to do was hit Connect. Screenshots:

The first screenshot shows what the media browser looks like with nothing plugged in. At that point, I plugged in my Creative Zen MicroPhoto, and you can see that it was detected and added to the device selector drop-down box:

After this, I hit the Connect button (it's the one on the top left) and artists were displayed. I hit the custom button just so you could see some of the details available:

So it's coming along. I expect this device detection to work on pretty much any kind of music player (except generic/vfat/UMS...more on that on a later post, when I have time to work on it), provided that:

1. hal is installed and working.
2. You are using a recent version of libmtp/libnjb for those devices. I've also worked with libkarma and libifp but have been unable to actually check them out to see how they're working since I don't have those devices. We're probably going to start a "send us your old media devices" thing going, so that I can work on all of them. libgpod was more hesitant about working with me, but there's a problem there anyways, as apparently you can't detect from the USB interface the iPod exposes what kind of iPod it actually is, without connecting to it. Feel free to blame Apple on this one, as it's probably on purpose in an attempt to keep you on iTunes. So there's a generic definition already in hal which should be (hopefully, pending testing, send me your old iPods) good enough to at least have it show up as an iPod. Once you click the connect button the library should be able to provide more info.
3. Your distribution's package of the device access libraries installs the hal fdi files provided by the libraries. For some unfathomable reason many (most?) don't -- even my beloved Gentoo. So start bugging your package maintainers early, because if you don't have these installed, you're not going to get autodetecting goodness. Some of these may end up in the main HAL distribution (I have repo access so can add them in) but for many it doesn't make sense as they're constantly being updated or changed, and it would require changing in two places instead of one. (Also, some libraries generate these automatically, instead of having a fixed file).

So that's it for now. The next step is to get manual adding of devices working again. And after that, dealing with generic devices in a smart way (the way will be Banshee-compatible too...thanks to Aaron Bockover for the excellent idea!) In the meantime, watch out for the Amarok Device Donation Program.

]]> http://jefferai.org/2007/10/go-solid-go/feed/ 11 http://jefferai.org/2007/06/the-real-first-khtml-browser-on-windows/ http://jefferai.org/2007/06/the-real-first-khtml-browser-on-windows/#comments Tue, 26 Jun 2007 01:35:03 +0000 jefferai http://blog.jefferai.org/?p=1029 There's been a lot of hubbub lately about Safari being released on Windows, which is based on WebKit, which is based on KHTML, and how it'll beat Konqueror as the first KHTML-based browser on Windows.

Then you hear this other camp firing back about Swift, whose homepage proudly declares "The First KHTML Browser for Windows."

However, I'm here to set the record straight. To the best of my knowledge the first KHTML-based browser on Windows is ThunderHawk, which appears to have been using KHTML under-the-hood since at least 2004 and probably at least 2002.

Yes, it's a WinCE/Windows Mobile browser. But it's still Windows.

]]> http://jefferai.org/2007/06/the-real-first-khtml-browser-on-windows/feed/ 11 http://jefferai.org/2007/05/device-handling-and-hal/ http://jefferai.org/2007/05/device-handling-and-hal/#comments Sat, 05 May 2007 19:01:35 +0000 jefferai http://blog.jefferai.org/?p=1026 My blog's been rather inactive for a while. Most of this has been because of being busy, and because of not doing hugely exciting things when I have had Amarok hacking time (mainly some bugfixes in stable branch). But now something's come along that warrants an entry.

Amarok's device handling is pretty adequate. Granted it could be better, but we support nearly every device on the market through some plugin or another. But there's one recurring problem: except in a few cases, Amarok doesn't know what device you have plugged in, and how to handle it. You have to tell this to Amarok. This means you, and we, have to deal with the manual creation of devices. Well, that's going to change.

Enter HAL, and its KDE4 paramour Solid. By now, you probably know what these do. HAL, Hardware Abstraction Layer, gathers information about your hardware. Solid reads it (and some other information) and outputs a KDE interface to it. Now by itself, this isn't enough, because HAL has a namespace called portable_audio_player, but, with a few exceptions, it doesn't actually say what type of device you're using -- simply "storage" for a USB Mass Storage interface, or "user" meaning that a userspace library needs to handle it.

However, for the last few weeks I've been working with HAL developers and the developers of libgpod, libifp, libkarma, libnjb, and libmtp. The goal has been to come up with a way that device access protocols can be auto-detected. If you know what access protocol a device uses, you can load the appropriate plugin automatically. Well, this has been done, and as a result the HAL portable_audio_player namespace spec is being modified to allow libraries to insert information they know about devices into HAL -- while at the same time keeping libraries from clobbering each other. Interested parties can find a draft version here, with the final verison going into HAL as soon as the slugs at freedesktop.org set up my account.

After this, I will be adding proper entries in libmtp and libnjb (easy, since I have commit access for both), making entries for libifp and libgpod, and working with the libkarma developers for their entries. After all this is done (probably in the next couple weeks), Amarok 2.0 and other HAL-aware clients should be able to determine and load the correct plugin for just about any device on the market, automatically. Okay...after a lot more hacking on 2.0...but you get the picture.

I'm hoping that when this is done we can remove manual device addition/deletion. It's a complex part of the code that can be difficult to maintain, and shouldn't be necessary any longer -- if you have a library that can handle the device that Amarok can use, Amarok should already know about the device from HAL. Even generic USB Mass Storage players should be fine, because they should be detected by HAL and exported through Solid as storage volumes. The only thing I can see this affecting negatively is the generic device plugin that just got fixed to support arbitrary KIO paths but I've been thinking about pulling that into the Collection some way anyways, now that the Collection can handle many sources, as this would let it handle *lots* of sources

I can't get database support working yet, but it does build and run. Check it out:

]]> http://jefferai.org/2007/02/amarok-on-mac/feed/ 17 http://jefferai.org/2006/05/bang-bam-boom/ http://jefferai.org/2006/05/bang-bam-boom/#comments Fri, 05 May 2006 22:52:49 +0000 jefferai http://blog.jefferai.org/?p=1024 I've been knocking down my to-do list like there's no tomorrow. I posted recently about the VFAT device rewrite; it is now complete and early testing results seem to be positive (it's been renamed to "Generic Audio Player" by the way). In addition, ATF backend support should now be fairly complete. I've been debugging it for a while and everything seems to work well now. It catches file copies, moves, and renames, and lets you assign new IDs to files through a DCOP call. It doesn't slow down scanning much either (I've been scanning 2082 files in about 30 seconds or so). I have yet to test it on Oggs, but I would imagine that it'd be fairly simple to get that working too.

The next hurdle is actually making amaroK components use ATF. I don't think that will happen in time for 1.4 final, but in the point releases thereafter hopefully you'll start to see some components take advantage of the features that ATF provides, to help it really live up to its name.

    The good news is that this time, I'm not hating it as much. This is mainly because I've started with a much more hierarchial design that makes tasks that were complex and error-prone in the old device code much more simple and reliable. The end result should be a device plugin that is much more stable and doesn't have as many weird quirks -- with any continuing quirks fixed quickly, of course!

    Today I got done with enough code to display the filesystem and allow for expanding directories. This may not sound like much, but it's actually a good sign, as these functions seem to work well and stably, and these basic functions being completed means that all the under-the-hood code is working (which was the majority of the rewrite), so the rest of the functions will be easy to implement. And the best news of all is that unlike the old code, I'm very confident in this new code!

]]> http://jefferai.org/2006/05/vfat-rewrite/feed/ 1 http://jefferai.org/2006/04/rip-your-cds-with-kio-and-amarok/ http://jefferai.org/2006/04/rip-your-cds-with-kio-and-amarok/#comments Mon, 24 Apr 2006 07:49:21 +0000 jefferai http://blog.jefferai.org/?p=1019     Yesterday I commited some code to the Collection Browser that allows it to accept URLs with the audiocd:/ protocol.  If your machine supports this (the audiocd:/ kioslave is probably in the kdemultimedia package), you can rip by doing the following;

1. Open KControl.  Go to Sound and Multimedia->Audio CDs.  Configure your ripping preferences.
2. Open Konqueror. Go to audiocd:/
3. Go into whatever folder you want your tracks encoded as, i.e. Ogg Vorbis will contain Vorbis "files" that you can select.
4. Drag these tracks to amaroK's Collection Browser.

amaroK will perform a KIO copy, which will result in a seamless rip of the track and import of the track into your collection.  Easy as pie, and using KDE's built-in capabilities to boot!