A/B testing small changes is something that really big sites like Amazon can do because they have enough volume to justify it. It's kind of like blood doping in sports. If you're already at the top of your game it will make enough of a difference to be significant, but if you're just some average person who can't run a mile... blood doping is the last thing you should do.
He said it with a grin because it is a huge amount in the context of a competitive time trial -- like the difference between first place and 10th or 20th place perhaps. But as a percentage difference it's roughly a 1.4% increase in speed (assuming you can maintain 50 km/h). It's practically nothing in real terms.
In competitive cycling, though, the margins are super small. You might win the Tour de France by 2 minutes, which seems like a pretty big lead, until you realise that's 2 minutes in 80 or 90 hours of cycling. This is why Team Sky's approach of "marginal gains" is so successful -- the difference between first and second place is something like 0.04% performance.
We've got this idea that we need to optimise performance (in terms of SEO, etc, etc), but I've never seen anybody quantify the margin of "victory" required. How much better do you need to get to push you over the edge? Because that's what's going to dictate what strategy you need to pursue.
The start-up failed for many reasons and this was one.
In a previous life I ran a Conversion Rate Optimization as a Service business and optimized $100s of millions of transactions for customers across categories.
The amount of research on this topic is staggering and a lot of it dates years back like Walmart's seminal review in 2012 (1) showing the devastating impact of speed on conversion rate.
In case that's not enough for you Google set up not one, but TWO separate page speed tools to help webmasters fix this chronic problem before they just threw in the towel and took over the job for you with their AMP effort. (2)
Oh and by the way did you know that if your site speed is slow Google will penalize your SEO? And drop your Page Quality Score index resulting in higher PPC charges?
The best technical guide on the internet to addressing this problem is here - http://httpfast.com/ - and a comprehensive monitoring service built by the author here - https://www.machmetrics.com/.
(1) http://www.webperformancetoday.com/2012/02/28/4-awesome-slid...
(2) https://testmysite.thinkwithgoogle.com/ and https://developers.google.com/speed/pagespeed/insights/