I really appreciate the walk through of the apache bench (ab) results and learning process even though it didn't get them to their objective - I've been thinking about using ab myself, and these are great things to know.

I took down a few notes while reading the article:

- mentions use of apache bench ( ab ) command for load testing

- mentions use of ganglia tool

- mentions configuring HAProxy for multi-core using nbproc setting

- mentions the 'parallel' tool for running commands in parallel

- simulate long run requests by having the server delay a little vs client (work around for ab deficiencies )

- have the server also send different response lengths back simulating varying load

- pdsh tool to remote parallel shell (ssh) sessions

- vegeta tool - that got them to their scalability / tipping point objective

- nodejs (used for their backends) had a default request timeout of 2 mins

- used dmesg to learn that haproxy was running out of mem (at around 1.2mm conns)

- pdsh to run vegeta tool on multiple machines (acting as clients) - script included in article

- mentions haproxy maxconn setting - and verification by checking proc fs limits for the haproxy pid