Most people have a single, important foreground app which is the only one they care about latency of. The others are ticking over in the background, but because they don't have window focus the user doesn't care about their responsiveness so much.