The benchmarking lie?

Let's write a few notes about benchmarking the different mp3 decoders, which are available. 'top' is NOT a benchmark, it's a simple check on how a program performs. The sad thing with 'top' (or better the linux kernel) is, that it has some problems with the measurement of threaded programs or programs only requesting short chunks of processor time. One real test is to measure how long your machine needs to decode a stream without threads with 100% CPU. Using mpg123 you can do this with

	time mpg123 -t mp3stream.mp3

or, if you additionally want to measure the I/O time:

	time mpg123 -s mp3stream.mp3  > /dev/null

If you find a player, which claims to be 10 or even more times faster than the current players: just don't believe it. A factor of 2 MAY BE possible.

Some tests results

This is using the benchmark-cpu Perl script, also present in the source tarball under scripts/. The user CPU time of mpg123 decoding the whole album Points of View by Convergence (get the tarball from here since Jamendo broke the link) has been measured. We are working on some nice presentation, for now you can have a look at the text files in the benchmark directory. If you want to add to this list, your contribution of the output of

	perl benchmark-cpu.pl /path/to/mpg123 convergence_-_points_of_view/*.mp3

is welcome, sent to the maintainer eMail or the user/dev mailing list (see contact section). To level the field, I tend to copy the MP3s to /dev/shm (RAM drive) first, skipping initial delay for reading from disk, but this shouldn't influence the measured user time anyway.