News for hackers. Stuff that matters.
Re:Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5... by sillysaurus on Sunday July 28, 2013 @08:11PM (#6118676)
Is there a domain where this is applicable outside of high performance trading? I'm trying to think of another use case that would legitimately generate 1M queries/sec.
Re:Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5... by derefr on Sunday July 28, 2013 @08:36PM (#6118765)
Some of the larger combats (5000+ ships) in EVE Online might do that.

(I single EVE out because other MMOs generally shard by user-cohort, so having that number of people on one shard is impossible. EVE, meanwhile, shards by location within the virtual world (each star system is a shard), so the entire player-base can "gather" on a single shard for a confrontation.)

Re:Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5... by recuter on Sunday July 28, 2013 @09:09PM (#6118857)
> In February 2013, EVE Online reached over 500,000 subscribers.

So what, the entire game userbase needs to be awake and in the same spot? :0)

Re:Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5... by ricardobeat on Sunday July 28, 2013 @10:22PM (#6119023)
Re:Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5... by voltagex_ on Sunday July 28, 2013 @09:46PM (#6118954)
Re:Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5... by pwelch on Sunday July 28, 2013 @09:14PM (#6118866)
I've never played EVE Online but from what I have read about it the engineering behind it seems interesting. Have any suggests for Dev Blog posts from their team?
Re:Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5... by colanderman on Sunday July 28, 2013 @08:50PM (#6118802)
Intrusion protection systems must legitimately handle upwards of 15M queries/sec when under attack (on a 10 Gbps link).
Re:Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5... by jonahx on Sunday July 28, 2013 @09:21PM (#6118880)
I'm curious, can you elaborate on this? What would they be querying? What kind of intrusion and what kind of attack?
Re:Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5... by colanderman on Sunday July 28, 2013 @10:33PM (#6119052)
TCP flow state, IP shunning & reputation, client rate limiting, others I'm probably missing.

Specifically during DDoS attacks, an IPS must usefully distinguish bad from good traffic at packet rates saturating a link. This inevitably involves maintaining lots of per-client and per-IP state. Caching is of little help precisely due to the distributed nature of the attack, and you can't shed load since that only helps the attacker.

In this situation, memory stalls become your biggest bottleneck – each one can eat on the order of 10% of your processing budget in a run-to-completion (RTC) design. The only solution (beyond tricksier data layouts) is memory latency hiding via micro- or hyperthreading (kernel context switches are just too slow). Rearchitecting a RTC design into a micro-threaded model is a lot of work, and bug-prone. Hyperthreading gives you latency hiding "for free", if the silicon supports it.

Generally you want between 2-4 micro- or hyperthreads. A second micro/hyperthread will generally just help keep the pipeline busy outside memory stalls; hence you can eke out extra performance with a third or a fourth. Intel chips only support two hyperthreads (when they do, and the OS supports it). Some more specialized processors support more.

Re:Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5... by jahaja on Sunday July 28, 2013 @08:26PM (#6118729)
A few off the points is applicable at much lower QPS than 1 million, especially in a virtualized environment such as EC2.

For example, I've had some issues with CPU (on core 0) saturation for soft interrupts on EC2, related to processing network packets.

Re:Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5... by stock_toaster on Sunday July 28, 2013 @08:27PM (#6118737)
Ad networks.