The way the data is being captured, is through: - Google Toolbar - Google Chrome - Google DNS services
Your observations of the speed and the improvements are not always similar as the ones from the aggregated data Google has access to.
I'm not sure what you are trying to accomplish with the dis-allowment of the next urls in robots.txt. Can you explain more what your hypothesis is in this test, and how you would measure success?
Perhaps a solution then would be to trigger the AJAX on mouse over, but that seems kludgy. In my case, I need make the AJAXed content part of the initial page load anyway, for the sake of user experience. But I can see cases where Google should not be counting AJAX as part of the page load time. God forbid somebody uses long polling for example. Maybe Google is doing this in a smart way, looking at the changes after the AJAX and determining if they should count as part of the page load.
You can experiment with Asynchronous calls, or slow load jQuery scripts, which kick off after the headers and html framework already have been loaded.
Overall, I would not worry about the reports in Google WMC that much, just try to get faster in general.
If you are serious in delivering an ultra speedy service online, there are services which can test your site or application on multiple locations, different OS and connection speed or using a different browser. But these services are pricy, trust me on that one!
For instance, let me introduce you to SETI@GoogleBot. SETI@GoogleBot is much like SETI@home except it takes advantage of GoogleBot's recently discovered capabilities. Including the SETI@GoogleBot script into your web pages will cause (after the page load event) the page to fetch a chunk of data from the SETI servers via ajax request and proceed to process that data in JavaScript. Eventually, once the data has been processed it will be posted back to the SETI servers (via another ajax request) and then repeat the cycle. Thus enabling you, for the small cost of a page load, have GoogleBot process your SETI data and enhance your SETI@home score.
Obviously, this isn't a new idea (using page loads to process data via JavaScript) but it is an interesting application to exploit GoogleBot's likely vast resources.