Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by dschiptsov on Sunday December 07, 2014 @01:53PM
(#8713125)
The "miracle" of Erlang/OTP is that it is well-researched and based on proper concepts, so there is absolutely no usual bullshitting here.
The crucial mantra by Joe Armstrong is "to model a behavior in a "parallel" real world one have to choice a proper abstraction (which is about granularity, not classes or objects), then everything follows".
So they have selected a connection as a fundamental abstractions and a lightweight process as a unit of granularity. Everything else follows, indeed.
Processes share nothing, so there is no locks, semaphores, race-conditions. There is even no loops - only recursive procedures, which could be restarted if crashed, or even its code could be replaced without stopping the service (why, a new copy of the code would be called "in a next recursive call", because it is stateless/pure-functional).
Each lightweight Erlang process serves its own connection, so it is perfectly natural and efficient to block/sleep on it (the OS kernel will do the job). It is not a "busy waiting", and there are no callbacks either. No "event buses", no nonsense.
There is no threads, so no shared state or even shared stack. If a process crashes (why should it? - it communicates with outside world only via message-passing) it cannot corrupt or lock any shared data, etc. The whole class of problems which comes form a flawed concept of pthreads does not arise.
There is no "reactive asynchronous event processing (written in Java or Scala)", there is no non-blocking I/O (btw, the only real non-blocking I/O could be done by an OS - aio_read/write and friends). There is no meaningless over-abstraction. That is why it works so well. Mo magic.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by perlgeek on Sunday December 07, 2014 @09:16AM
(#8712209)
This feels like mixing of levels to me. The Internet traffic of nearly every user will run through a Cisco or Juniper router. Does that make us all IOS or JunOS users?
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by justincormack on Sunday December 07, 2014 @08:15AM
(#8712055)
Netflix is the other FreeBSD based service that many people dont know about. While the control plane is the much hyped Linux on AWS stuff, the actual streams are FreeBSD boxes in ISPs[1].
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by justincormack on Sunday December 07, 2014 @11:38AM
(#8712656)
I would only really count Juniper in that list, building on and with FreeBSD. Apple take some bits but their product is not really FreeBSD (alas) and they dont seem to give back. AFAIK EMC and Yahoo are ex-users now, or largely.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by feld on Sunday December 07, 2014 @01:13PM
(#8712995)
EMC Isilon still uses FreeBSD. There are employees who are also FreeBSD developers. They're working hard on giving back changes so they can jump to a much newer FreeBSD release.
Yahoo's FreeBSD isn't dead; YBSD! still exists too.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by praveenster on Sunday December 07, 2014 @10:05AM
(#8712347)
Since Whatsapp doesn't have client facing web page, but only api's used by the client apps, are their scalability requirements different from those that have a web interface as well?
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by Demiurge on Sunday December 07, 2014 @11:40AM
(#8712663)
messages are probably much smaller, but more numerous and latency has to be 'realtime', but also client-server-client juggling. so, to me this means they need to respond faster, more often, and deal with more sharing than a website.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by jambo on Sunday December 07, 2014 @12:04PM
(#8712756)
I found the pragmatism and sense of deferring until the least responsible moment—especially the speaker's view of what that moment is—very interesting.
In the Q&A "What do you use for configuration management?" "<slight laugh> `make`", going on to say they don't have enough node types or a large enough cluster to do automation yet.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by tachion on Sunday December 07, 2014 @07:17AM
(#8711957)
The servers are FreeBSD 9.x. After Facebook acquisition they've tried migrating it all to Linux but quickly discovered that Linux cant cope with that amount of traffic and they decided the FreeBSD is the tool to be used here. That lead to 'famous' Facebook job ad looking for Linux kernel engineers to help them bring the Linux TCP/IP capabilities in par with FreeBSD. Currently they're testing FreeBSD 10.1-RELEASE for their next platform OS upgrade.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by atmosx on Sunday December 07, 2014 @08:13AM
(#8712051)
Do you have any idea why Facebook would rather develop a TCP/IP Stack for Linux to match FreeBSD's, instead of further evolving FreeBSD's stack? Licensing issues?
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by tachion on Sunday December 07, 2014 @10:56AM
(#8712510)
While I have no actual knowledge why they made that decision, its not very hard to guess, and the possible answers are around their own staff familiarity with FreeBSD, the pros of having the platform simplified by using single OS (Linux in their case, since for sure they've more servers with Linux already than they've acquired with WhatsApp) and all possible security and financial implications following these factors. However, it seems like at least for know they might be staying with FreeBSD a bit longer, what's a great PR/recognition win for the FreeBSD itself. I really doubt its for 'licencing issues' because (religious wars aside) the BSD license is way more attractive to businesses than GPL one.
Said all that - I might be terribly wrong about everything I said, because its not an insider knowledge ;)
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by justincormack on Sunday December 07, 2014 @08:19AM
(#8712060)
How could licensing issues be a problem?
I think people prefer a monoculture, even though the security idea of having multiple software stacks is appealing, thoroughly testing issues on two OS stacks is a lot more work, especially at Facebook scale.
Presumably they are tied into Linux for a lot of their internal infrastructure.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by justincormack on Sunday December 07, 2014 @10:09AM
(#8712356)
If someone getting into your network is your threat model then, well, you have already been owned. If you want to have resilient services where you can swap them out if there are security issues, a monoculture is not a win.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by ps0ps on Sunday December 07, 2014 @11:40AM
(#8712666)
This is not true. Facebook has never attempted to migrate them to Linux. If WhatsApp wants to convert to Linux, it is their decision. More than likely they will stay on FreeBSD as that is what they know how to run and a good way to kill a product is to force them to do things instead of focusing on building their product.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by wslh on Sunday December 07, 2014 @08:14AM
(#8712054)
> Linux cant cope with that amount of traffic
I like FreeBSD but I don't think we can say that Linux can't cope with that amount of traffic while Google and Facebook use Linux. There is a missing piece here that's not about network traffic.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by justincormack on Sunday December 07, 2014 @08:21AM
(#8712065)
I think it is amount of traffic per server. WhatsApp didnt have a lot of servers and maybe reengineering for massive scale out on Linux doesnt make sense versus staying with what works.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by jlouis on Sunday December 07, 2014 @10:01AM
(#8712336)
It doesn't have to be a weaker stack per se. It is also a lot about knowledge. WhatsApp vested much time into understanding FreeBSD. It may be if you vested the equal amount of time on Linux with knowledgable people, then you would see roughly the same performance.
If I should take a guess, the advantage of the FreeBSD stack is that it is simpler and easier to understand. Hence, it is easier to tune for performance.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by andor on Sunday December 07, 2014 @08:25AM
(#8712073)
Yes, of course, and Whatsapp users care very much about what's going on behind the scenes ;-) I'm not a mainframe user because I book flights, or a Windows user because i tried Bing.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by sandGorgon on Sunday December 07, 2014 @01:33PM
(#8713055)
Correct me if I'm wrong, but one of the biggest scaleups in FreeBSD seemed to be zero-copy sockets. This should be solved in Linux when kdbus is turned on (replacing the current IPC : D-Bus).
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by sandGorgon on Sunday December 07, 2014 @01:01PM
(#8712951)
if one HAD to choose a non-erlang stack (for whatever reason), what would you choose ?
Go, Nodejs or Java - I'm mainly concerned about clustering, because I think that is what gave the Whatsapp devs their ability to scale(without having to do too much re-engineering).
Or can the clustering be moved to something like the queuing layer ?
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by sandGorgon on Sunday December 07, 2014 @01:28PM
(#8713038)
I wsa hoping you wouldnt say that :(
Java high performance clustering is... intimidating. It is too enterprisey. From my (little) research, it seems that one would need to build this ground up on top of Netty or Undertow.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by marktangotango on Sunday December 07, 2014 @02:20PM
(#8713230)
Care to elaborate? Load balancing to a bunch of http servers, sticky sessions, or store the session to a db, or the best option is no session data, memcached, its all pretty straight forward?
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD...
by dschiptsov on Sunday December 07, 2014 @11:39AM
(#8712662)
Showing FreeBSD/Erlang stuff that just works and cope with substantial amounts of traffic is not politically correct, because it will cause a severe pain of cognitive dissonance to the Docker/JVM/Node/Cloud crowds..
Lets chant together: JVM is the best VM in the world with millions of man-hours invested in optimizations. Servlet/JSP are mature, battle-tested, enterprise-grade, type-safe solutions. NodeJS is the only way to develop high-scalable internet services. Everything must be a containerized micro-service, orchestrated by Docker/CoreOS/systemd.
http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...
The crucial mantra by Joe Armstrong is "to model a behavior in a "parallel" real world one have to choice a proper abstraction (which is about granularity, not classes or objects), then everything follows".
So they have selected a connection as a fundamental abstractions and a lightweight process as a unit of granularity. Everything else follows, indeed.
Processes share nothing, so there is no locks, semaphores, race-conditions. There is even no loops - only recursive procedures, which could be restarted if crashed, or even its code could be replaced without stopping the service (why, a new copy of the code would be called "in a next recursive call", because it is stateless/pure-functional).
Each lightweight Erlang process serves its own connection, so it is perfectly natural and efficient to block/sleep on it (the OS kernel will do the job). It is not a "busy waiting", and there are no callbacks either. No "event buses", no nonsense.
There is no threads, so no shared state or even shared stack. If a process crashes (why should it? - it communicates with outside world only via message-passing) it cannot corrupt or lock any shared data, etc. The whole class of problems which comes form a flawed concept of pthreads does not arise.
There is no "reactive asynchronous event processing (written in Java or Scala)", there is no non-blocking I/O (btw, the only real non-blocking I/O could be done by an OS - aio_read/write and friends). There is no meaningless over-abstraction. That is why it works so well. Mo magic.
A few German game studios are using it on their backends.
http://de.slideshare.net/wooga/erlang-the-big-switch-in-soci...
http://sdtimes.com/enterprises-can-learn-from-game-developme...
You could say the same about erlang for example, it used by WhatsApp and in many internet-cell network gateways.
[1] https://openconnect.itp.netflix.com/software/index.html
Yahoo's FreeBSD isn't dead; YBSD! still exists too.
In the Q&A "What do you use for configuration management?" "<slight laugh> `make`", going on to say they don't have enough node types or a large enough cluster to do automation yet.
Said all that - I might be terribly wrong about everything I said, because its not an insider knowledge ;)
I think people prefer a monoculture, even though the security idea of having multiple software stacks is appealing, thoroughly testing issues on two OS stacks is a lot more work, especially at Facebook scale.
Presumably they are tied into Linux for a lot of their internal infrastructure.
In fact running multiple operating systems is more likely to lead to a misconfiguration issue which allows an attacker into your network.
Similarly one of the reasons Facebook relies on PHP is because a lot of developers are familiar with it.
I like FreeBSD but I don't think we can say that Linux can't cope with that amount of traffic while Google and Facebook use Linux. There is a missing piece here that's not about network traffic.
If I should take a guess, the advantage of the FreeBSD stack is that it is simpler and easier to understand. Hence, it is easier to tune for performance.
[1] https://www.starve.io
Go, Nodejs or Java - I'm mainly concerned about clustering, because I think that is what gave the Whatsapp devs their ability to scale(without having to do too much re-engineering).
Or can the clustering be moved to something like the queuing layer ?
There are lots of JVM implementations to choose from, many with high quality code generation (both AOT and JIT) and GC implementations.
It has VM and cluster monitoring tools only comparable by .NET offerings.
The majority of big data solutions are built on top of it.
There are lots of libraries to choose from.
If Java 8, the language, isn't your thing, there are lots of modern languages to choose from.
Java high performance clustering is... intimidating. It is too enterprisey. From my (little) research, it seems that one would need to build this ground up on top of Netty or Undertow.
Lets chant together: JVM is the best VM in the world with millions of man-hours invested in optimizations. Servlet/JSP are mature, battle-tested, enterprise-grade, type-safe solutions. NodeJS is the only way to develop high-scalable internet services. Everything must be a containerized micro-service, orchestrated by Docker/CoreOS/systemd.