Re:In search of performance – how we shaved 200ms off every ...
by dap on Thursday August 20, 2015 @05:52PM
(#10094878)
It's not that you can't find _some_ differences that way. As others have you pointed out, there should be little difference between the software deployed in staging and production. But there are can be differences in performance and hardware configuration (unless you can afford to mirror your production deployment exactly). Most unavoidably, there are problems you only hit after cumulative amounts of uptime or load[1], which is nearly impossible to simulate in a preproduction environment.
[1] For an example, see our recent outage related having seen 200M PostgreSQL transaction: https://www.joyent.com/blog/manta-postmortem-7-27-2015
Re:In search of performance – how we shaved 200ms off every ...
by edwhitesell on Friday August 21, 2015 @09:39AM
(#10097321)
Fair enough. I guess my point should have been: With proper tracking of changes and work done between the environments, finding differing versions could be the lowest hanging fruit.
That was my response to your original question of how they found got to haproxy as the root cause so quickly.
Sure, mirroring hardware and performance can help, if you can afford it (which is rare). However, you'd also probably agree it's pretty rare to find an issue caused cumulative amounts of uptime or load. Mis-matched software versions or configurations are a far more likely culprit in my experience.
Horses not zebras...
If you don't have processes and documentation to quickly point out those kinds of differences between environments, you're doing something wrong.