News for hackers. Stuff that matters.
Re:How We Went from 30 Servers to 2: Go by waterside81 on Tuesday March 12, 2013 @08:43PM (#5366046)
And that's what we did during our first pass: identify the slow parts and port them to C. This gave us about 30% better performance. Without getting too much into the nitty gritty of our specific case, this wasn't enough.

We needed some massive speed improvements, I'm talking in the order of 100X faster. The nature of our algorithms was such that they could be done in parallel (i.e map/reduce) - an ideal candidate for Go's goroutines. We actually tried to make it parallelizable at first in Python, using gevent (and even just multiprocessing) and the results were not great.

One other aspect that really guided us towards Go was memory usage. Python was just sucking up so much memory whereas our Go implementation thus far is so much thinner.

Re:How We Went from 30 Servers to 2: Go by danieldk on Wednesday March 13, 2013 @05:53AM (#5367604)
One other aspect that really guided us towards Go was memory usage. Python was just sucking up so much memory whereas our Go implementation thus far is so much thinner.

It's funny that you mention that, because in computationally intensive work memory management is an issue with Go. E.g. I wrote a maximum entropy parameter estimator in Go, which was terribly slow until I circumvented the garbage collector by preallocating a huge block of memory and doing my own management. In C malloc() and free()-ing had practically no overhead. After putting the Go garbage collector out of the game, the Go version was approximately within 2x of the C version.

I am interested how Go gave you one or two orders of magnitude speedup, while rewriting hot spots in C didn't...

Re:How We Went from 30 Servers to 2: Go by Ziomislaw on Wednesday March 13, 2013 @06:16AM (#5367658)
> I am interested how Go gave you one or two orders of magnitude speedup, while rewriting hot spots in C didn't...

paralellization

Re:How We Went from 30 Servers to 2: Go by danieldk on Wednesday March 13, 2013 @09:28AM (#5368194)
One can easily do that in C with OpenMP.
Re:How We Went from 30 Servers to 2: Go by Symmetry on Wednesday March 13, 2013 @11:14AM (#5369011)
If you can get away with only parallelizing the C code that'll work, but if you're replacing individual Python functions calls it won't so much. You could use a lot of Python processes, but that might be awkward depending on how much they need to coordinate and might lead to too much memory usage.