By getting statistics from the scheduler itself: how long did the process have to wait before it started executing. Also by looking at global effects: run N mplayers at the same time and look for dropped frames ; run an app that maps similar amount of work to N children and measure the run time ; measure the time needed for compiling [big project] with concurency == number of cpus ; ...
These are both the kind of tasks Kolivas is willing to totally throw under the bus for better worst case latency.
Remember, the single goal of the project is to get worst-case latency for all non-cpu-limited processes on the system as low as it possibly can get. Throughput, and even best- and average case latency are considered entirely expendable. This is why the BFS is hard to benchmark.
If you're purporting to improve the performance of something, but you aren't always measuring the impact your changes have on the actual performance of that something, you're working blind.
"For years we've been doing our workloads on linux to have more work than we had CPUs because we thought that the "jobservers" were limited in their ability to utilise the CPUs effectively (so we did make -j6 or more on a quad core machine for example). This scheduler proves that the jobservers weren't at fault at all, because make -j4 on a quad core machine with BFS is faster than any choice of job numbers on CFS." [1]
I've just been listing to lectures from MIT's introduction to algorithms, where more emphasis is placed on worst-case scenarios than otherwise, the reasoning being that these make guarantees possible. So more power to BFS.
Yes you can.
And you answered your own question: latency is exactly how you benchmark this (with a measure of total throughput added in to make sure it's not low).
At some point in the discussion, someone introduces a benchmark that clearly reproduces a problem with the current CFS scheduler. I think that perfectly illustrates what you are looking for (unfortunately, I have to run and can't pinpoint the exact post in this timeframe).
* kernel build
* pipe
* messaging performance
* OLTP performance
And then wonders why he doesn't get an improvement?
One of the other posters puts it perfectly:
"Do you know what a normal desktop PC looks like? No, a more realistic question based on what you chose to benchmark to prove your point would be: Do you know what normal people actually do on them?"
In answer to the parent poster, see Nikos Chantziaras' posts.
(FWIW, my 1.2GHz PPC running Debian performs unacceptably as a desktop machine. There are sometimes >1000ms lags between a keypress and it being reflected on the X11 display with no background load. But I can't say this is because of the scheduler.)