As measured it was actually talking about total round-trip time including server processing, but it was measured from within the JS on the client, from start-of-AJAX-request to finish, and so it included both server/client hops in addition to processing time. That's part of the red herring: since you expect the bottleneck to be server processing time, you look there first.

A decent partial answer to "How to prevent this?" is to look at additional metrics - graphing server-only latency would show the expected correlation with server load and an inverse correlation with total RTT, and would narrow the search space significantly.