News for hackers. Stuff that matters.
Re:How We Went from 30 Servers to 2: Go by orangethirty on Tuesday March 12, 2013 @06:44PM (#5365485)
What changed in the architecture? It is rather interesting that by just having a language change you could remove 28 servers from the system.
Re:How We Went from 30 Servers to 2: Go by ikeepforgetting on Tuesday March 12, 2013 @06:54PM (#5365544)
My guess is that removing 2-5 layers of abstraction can get you pretty far along.
Re:How We Went from 30 Servers to 2: Go by buro9 on Tuesday March 12, 2013 @07:11PM (#5365634)
We're writing most of the systems for our company in Go, and of the many reasons this ranked highly. Just removing the multiple unnecessary abstractions gets you a hell of a lot of simplicity and performance back.
Re:How We Went from 30 Servers to 2: Go by ikeepforgetting on Tuesday March 12, 2013 @07:36PM (#5365754)
I work in an area that requires a lot of high performance computing and there is a reason people in this area like code simplicity. Stacking layer upon layer of complex algorithms is a lot more difficult to reason about than having a simple and elegant system that's close to the metal.

There is a recent trend of building more and more complex systems, have less coupling and 'good oop'. That's all good and nice but simple code is:

  - more maintainable (a complex nest of objects & function calls all loosely coupled isn't)
  - a lot more readable
  - easier to optimize
  - better for your sanity
Some languages have build a culture of extreme complexity and are notable for building huge structures and systems to do simple tasks. Having a 50K lines of code dedicated to just inversion of control or O/R mapping makes a problem a lot more complex than it probably needs to be.
Re:How We Went from 30 Servers to 2: Go by martinced on Tuesday March 12, 2013 @08:41PM (#5366033)
"Having a 50K lines of code dedicated to just inversion of control or O/R mapping makes a problem a lot more complex than it probably needs to be."

Exactly. That's the terrible stuff with all this "enterprisey" mindset: people are working on medium-sized codebase made of this special kind of hell that Java/C# + ORM ([N]hibernate) + XML + SQL is and these app often run into the 200K / 300 KLOC lines if not more. Yet what does these applications really do? Actually not very much. Yet these programmers are sure to be working on super-advanced stuff because their codebase is big.

When several companies reported a drop in LOC of 90% by switching to something else than Java then at one point you have to at least consider that maybe most of your Java codebase is hot air.

Re:How We Went from 30 Servers to 2: Go by FooBarWidget on Tuesday March 12, 2013 @07:49PM (#5365818)
Abstractions are not necessarily slow. The BSD socket API is an abstraction over sending raw TCP/IP packets to your Ethernet card, but you're not going to get a whole lot more performance out of accessing your Ethernet card directly.
Re:How We Went from 30 Servers to 2: Go by ikeepforgetting on Tuesday March 12, 2013 @08:00PM (#5365862)
Well, they're always slower than not having them. Having a few layers on top of each other starts to add up. It's very easy to lose track of the bottlenecks of 2 or more layers below the one you're working in. There is a reason HTTP.sys runs in kernel space and BSD sockets don't. And there's a reason both of them are slower than XMLHttpRequests.

PS. The fact that you can point to one (or a few) efficient abstractions doesn't mean they all are.

Re:How We Went from 30 Servers to 2: Go by taeric on Tuesday March 12, 2013 @09:41PM (#5366267)
Just no. Some abstractions are light years faster than not having them. To the point that this is a silly statement. Consider the elementary abstraction of an array. The slightly less elementary one of a map.

More close to heart, the abstraction that source control gives you. Imagine not having git (or whatever tool) and thinking you will compare this weeks code with what was available last week. Now just do the same based on the changes introduced by a single developer. Trivial with the abstraction of source control. Hella tough without.

So, no, just because there are layers of abstractions does not automatically mean things are going to be slow.

All of that said, I can see and ultimately agree with your point. Excessive abstraction can be bad. Unfortunately, I think the jury is out on where the hell that line is. (Likely they are distracted with other pointless questions while others are out solving problems.)

Re:How We Went from 30 Servers to 2: Go by iforgotagain on Wednesday March 13, 2013 @06:24AM (#5367673)
>> Just no. Some abstractions are light years faster than not having them. To the point that this is a silly statement. Consider the elementary abstraction of an array. The slightly less elementary one of a map.

Abstractions can definitely be an improvident in some cases. They just bring extra complexity at it's cost. In the 'map' case you pointed out, you now need to take into account the performance characteristics of an array, one or more hash functions and linked lists in their specific configuration.

Even though the lookup of a hashmap is supposed to be O(1), you now have to run a hash function for every lookup. That's bound to be slower than the simple integer addition we were doing before on the flat array.

Besides that, if it's a bad hash function (or if the data is bad), performance will easily go down to O(n).

>> So, no, just because there are layers of abstractions does not automatically mean things are going to be slow.

That's not what I'm saying is it? I'm making the point that it'll be slow/er/ because of the overhead of the abstraction. I didn't say "less productive" or "every abstraction is bad", I'm point out that each extra layer comes at a cost and that the extra cost gets buried the more layers you add.

I'm not going to go into the source control argument because I'm talking runtime application performance and mental overhead, not developer performance.

Re:How We Went from 30 Servers to 2: Go by taeric on Wednesday March 13, 2013 @08:31AM (#5367972)
But that is my point, if used correctly, the abstraction just makes it easier to do what you want to do quickly. Consider, math is basically abstracting away the tedious nature of some physical process. I mean, if you know it takes 2 eggs to make 4 waffles, and want to make 82 waffles, the abstraction of math makes this much quicker than just doing it to determine how many eggs you need.

Basically, I object to the idea that abstractions slow you down by their nature. I agree that poor or unnecessary ones do so.

Re:How We Went from 30 Servers to 2: Go by qznc on Wednesday March 13, 2013 @03:07AM (#5367231)
I chuckled at "just a language change".

What could be more expensive than rewriting all you code? For many Rails web apps the cost of additional servers is far below the cost to rewrite things in Go.

Re:How We Went from 30 Servers to 2: Go by fauigerzigerk on Wednesday March 13, 2013 @05:24AM (#5367536)
The cost of running 28 servers is not negligible compared to developer time. If a server goes for, say, $150 per month that's $4200 wasted each month. And contrary to the rewrite, the server cost is perpetual, so the rewrite amortises eventually.

Obviously, the economics of a rewrite depend very much on the size and complexity of the code base, but it didn't sound like they had a huge code base at the time of the rewrite.

Re:How We Went from 30 Servers to 2: Go by orangethirty on Wednesday March 13, 2013 @03:36AM (#5367300)
Thus the improvements where not a result of using Go (which is is a fine language). The improvements came from replacing a bloated system with a leaner one. You can do that with many other languages. There is no magic in that.