Ah, I see now... I didn't realize that this was end-user data via Google Toolbar opt-in. I've got "medium accuracy" 100-1000 data points, and I'm assuming that means total over the full 5 months. In that case, it doesn't surprise me that some combinations of visitors and connections might have long load times. But yes, the fact that non-visible elements seem to make a difference is not great.
The worst offenders to the speed of my site is that I have both google adsense and google analytics on it. I find it ironic.
From what I understand the time is sampled from visitors, and includes the time from the page request until the page is fully rendered.
I'd love to know what people do that get their times well below sub-second.