Speaking as a user, when someone sends me a reply, I expect to see it in seconds via the streaming API so I can have a short IM-style conversation, and I consider this a nice benefit to using Twitter (rather than, say, email), if not totally essential. Hundreds of seconds sounds bad, but if that were the norm, it would be a serious problem.
I think the real problem is not latency, but simply they don't have the signals needed to differentiate spam from not.