News for hackers. Stuff that matters.
Re:Keeping Instagram up with over a million new users in twe... by lenn0x on Thursday April 05, 2012 @10:31PM (#3805696)
What kind of instances are you guys running for Redis/memcached? I am a bit surprised on the numbers here, but to be fair I don't do much in the virtualization world. With low cpu overhead, it sounds like you might be saturating the number of interrupts on the network card if it's not a bandwidth issue. Memcache can usually push 100-300k/s on an 8-core Westmere (could go higher if you removed the big lock). Redis on the other hand with pinned processes to each physical core can do about 500,000/s. We (Twitter) saw saturation around 100,000~ on CPU0, what tipped us off was ksoftirq spinning at 100%. If you have a modern server and network card, just pin each IRQ for every TX/RX queue to an individual physical core.
Re:Keeping Instagram up with over a million new users in twe... by mikeyk on Thursday April 05, 2012 @11:14PM (#3805824)
Those are really useful numbers--I think a lot of it can be chalked up to virtualization, but we should definitely explore more around IRQ pinning for queues. Any good starting points / reading, are you mostly using taskset?
Re:Keeping Instagram up with over a million new users in twe... by lenn0x on Friday April 06, 2012 @12:30AM (#3806012)
Taskset is fine for the process pinning. Don't forget about hyperthreading, you want to try to keep each thread on each hardware thread. IRQ pinning, see an example script I have:

http://chrisgoffinet.com/pin_network.sh

This will set queues 0-7 to specific smp affinity slots.

Re:Keeping Instagram up with over a million new users in twe... by rkurian on Tuesday April 10, 2012 @03:40AM (#3821211)
Why do you guys use both memcache and Redis ? Redis also has LRU cache functionality.
Re:Keeping Instagram up with over a million new users in twe... by lenn0x on Sunday April 15, 2012 @01:56AM (#3842938)
Because in-house we have a custom version of memcache. We rewrote memcache's slab allocator, and for some use cases, is better at memory efficiency than Redis.