Re:Keeping Instagram up with over a million new users in twe...
by lenn0x on Thursday April 05, 2012 @10:31PM
(#3805696)
What kind of instances are you guys running for Redis/memcached? I am a bit surprised on the numbers here, but to be fair I don't do much in the virtualization world. With low cpu overhead, it sounds like you might be saturating the number of interrupts on the network card if it's not a bandwidth issue. Memcache can usually push 100-300k/s on an 8-core Westmere (could go higher if you removed the big lock). Redis on the other hand with pinned processes to each physical core can do about 500,000/s. We (Twitter) saw saturation around 100,000~ on CPU0, what tipped us off was ksoftirq spinning at 100%. If you have a modern server and network card, just pin each IRQ for every TX/RX queue to an individual physical core.
[ Parent
| on HN ]
Re:Keeping Instagram up with over a million new users in twe...
by mikeyk on Thursday April 05, 2012 @11:14PM
(#3805824)
Those are really useful numbers--I think a lot of it can be chalked up to virtualization, but we should definitely explore more around IRQ pinning for queues. Any good starting points / reading, are you mostly using taskset?
Re:Keeping Instagram up with over a million new users in twe...
by lenn0x on Friday April 06, 2012 @12:30AM
(#3806012)
Taskset is fine for the process pinning. Don't forget about hyperthreading, you want to try to keep each thread on each hardware thread. IRQ pinning, see an example script I have:
http://chrisgoffinet.com/pin_network.sh
This will set queues 0-7 to specific smp affinity slots.