News for hackers. Stuff that matters.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by wallflower on Sunday December 07, 2014 @09:37AM (#8712268)
To me, the more fascinating fact is that Erlang enabled their massive ability to scale.

http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by dschiptsov on Sunday December 07, 2014 @01:53PM (#8713125)
The "miracle" of Erlang/OTP is that it is well-researched and based on proper concepts, so there is absolutely no usual bullshitting here.

The crucial mantra by Joe Armstrong is "to model a behavior in a "parallel" real world one have to choice a proper abstraction (which is about granularity, not classes or objects), then everything follows".

So they have selected a connection as a fundamental abstractions and a lightweight process as a unit of granularity. Everything else follows, indeed.

Processes share nothing, so there is no locks, semaphores, race-conditions. There is even no loops - only recursive procedures, which could be restarted if crashed, or even its code could be replaced without stopping the service (why, a new copy of the code would be called "in a next recursive call", because it is stateless/pure-functional).

Each lightweight Erlang process serves its own connection, so it is perfectly natural and efficient to block/sleep on it (the OS kernel will do the job). It is not a "busy waiting", and there are no callbacks either. No "event buses", no nonsense.

There is no threads, so no shared state or even shared stack. If a process crashes (why should it? - it communicates with outside world only via message-passing) it cannot corrupt or lock any shared data, etc. The whole class of problems which comes form a flawed concept of pthreads does not arise.

There is no "reactive asynchronous event processing (written in Java or Scala)", there is no non-blocking I/O (btw, the only real non-blocking I/O could be done by an OS - aio_read/write and friends). There is no meaningless over-abstraction. That is why it works so well. Mo magic.

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by rudiger on Sunday December 07, 2014 @11:03AM (#8712536)
Not just Erlang/BEAM optimization, but the team's background in high performance, C-based systems on FreeBSD enabled them to achieve massive scale.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by Ixiaus on Sunday December 07, 2014 @11:05AM (#8712545)
What's more fascinating is that erlang underpins many more staples of the internet than most geeks even know. WhatsApp is just one 19B example.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by jambo on Sunday December 07, 2014 @12:00PM (#8712739)
Yep. Especially when you consider that things like RabbitMQ, Riak, ejabberd, etc. are written in Erlang and take advantage of OTP.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by pjmlp on Sunday December 07, 2014 @01:22PM (#8713029)
Not only the Internet.

A few German game studios are using it on their backends.

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by marktangotango on Sunday December 07, 2014 @02:09PM (#8713190)
Physics in Erlang? Game scripting engine implemented in Erlang? What types of games are we talking about here?
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by pjmlp on Sunday December 07, 2014 @02:44PM (#8713309)
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by perlgeek on Sunday December 07, 2014 @09:16AM (#8712209)
This feels like mixing of levels to me. The Internet traffic of nearly every user will run through a Cisco or Juniper router. Does that make us all IOS or JunOS users?
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by rdtsc on Sunday December 07, 2014 @10:08AM (#8712353)
It will and if Junos or ios stop working so will the internet to a large extent. It speaks to reliability and scalability aspects.

You could say the same about erlang for example, it used by WhatsApp and in many internet-cell network gateways.

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by justincormack on Sunday December 07, 2014 @08:15AM (#8712055)
Netflix is the other FreeBSD based service that many people dont know about. While the control plane is the much hyped Linux on AWS stuff, the actual streams are FreeBSD boxes in ISPs[1].

[1] https://openconnect.itp.netflix.com/software/index.html

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by tachion on Sunday December 07, 2014 @10:58AM (#8712521)
There's also Juniper, EMC2, Apple, Yahoo (back in the day, at least), Sony PlayStation and many, many others.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by justincormack on Sunday December 07, 2014 @11:38AM (#8712656)
I would only really count Juniper in that list, building on and with FreeBSD. Apple take some bits but their product is not really FreeBSD (alas) and they dont seem to give back. AFAIK EMC and Yahoo are ex-users now, or largely.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by feld on Sunday December 07, 2014 @01:13PM (#8712995)
EMC Isilon still uses FreeBSD. There are employees who are also FreeBSD developers. They're working hard on giving back changes so they can jump to a much newer FreeBSD release.

Yahoo's FreeBSD isn't dead; YBSD! still exists too.

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by adamnemecek on Sunday December 07, 2014 @01:16PM (#8713005)
Also HN :-)
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by praveenster on Sunday December 07, 2014 @10:05AM (#8712347)
Since Whatsapp doesn't have client facing web page, but only api's used by the client apps, are their scalability requirements different from those that have a web interface as well?
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by Demiurge on Sunday December 07, 2014 @11:40AM (#8712663)
messages are probably much smaller, but more numerous and latency has to be 'realtime', but also client-server-client juggling. so, to me this means they need to respond faster, more often, and deal with more sharing than a website.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by jambo on Sunday December 07, 2014 @12:04PM (#8712756)
I found the pragmatism and sense of deferring until the least responsible moment—especially the speaker's view of what that moment is—very interesting.

In the Q&A "What do you use for configuration management?" "<slight laugh> `make`", going on to say they don't have enough node types or a large enough cluster to do automation yet.

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by general_failure on Sunday December 07, 2014 @05:28AM (#8711820)
Not sure the title is correct. After all, the people use WhatsApp through Linux (Android), Symbian or whatever and not FreeBSD.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by adamnemecek on Sunday December 07, 2014 @05:43AM (#8711837)
The servers were (are?) FreeBSD.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by tachion on Sunday December 07, 2014 @07:17AM (#8711957)
The servers are FreeBSD 9.x. After Facebook acquisition they've tried migrating it all to Linux but quickly discovered that Linux cant cope with that amount of traffic and they decided the FreeBSD is the tool to be used here. That lead to 'famous' Facebook job ad looking for Linux kernel engineers to help them bring the Linux TCP/IP capabilities in par with FreeBSD. Currently they're testing FreeBSD 10.1-RELEASE for their next platform OS upgrade.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by atmosx on Sunday December 07, 2014 @08:13AM (#8712051)
Do you have any idea why Facebook would rather develop a TCP/IP Stack for Linux to match FreeBSD's, instead of further evolving FreeBSD's stack? Licensing issues?
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by tachion on Sunday December 07, 2014 @10:56AM (#8712510)
While I have no actual knowledge why they made that decision, its not very hard to guess, and the possible answers are around their own staff familiarity with FreeBSD, the pros of having the platform simplified by using single OS (Linux in their case, since for sure they've more servers with Linux already than they've acquired with WhatsApp) and all possible security and financial implications following these factors. However, it seems like at least for know they might be staying with FreeBSD a bit longer, what's a great PR/recognition win for the FreeBSD itself. I really doubt its for 'licencing issues' because (religious wars aside) the BSD license is way more attractive to businesses than GPL one.

Said all that - I might be terribly wrong about everything I said, because its not an insider knowledge ;)

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by justincormack on Sunday December 07, 2014 @08:19AM (#8712060)
How could licensing issues be a problem?

I think people prefer a monoculture, even though the security idea of having multiple software stacks is appealing, thoroughly testing issues on two OS stacks is a lot more work, especially at Facebook scale.

Presumably they are tied into Linux for a lot of their internal infrastructure.

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by Perdition on Sunday December 07, 2014 @09:12AM (#8712195)
The main security threat these days isn't viruses or worms so a monoculture makes no real difference.

In fact running multiple operating systems is more likely to lead to a misconfiguration issue which allows an attacker into your network.

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by justincormack on Sunday December 07, 2014 @10:09AM (#8712356)
If someone getting into your network is your threat model then, well, you have already been owned. If you want to have resilient services where you can swap them out if there are security issues, a monoculture is not a win.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by sciurus on Sunday December 07, 2014 @01:06PM (#8712964)
Facebook is almost certainly further evolving Linux's existing TCP/IP stack, not developing a new one.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by MasterScrat on Sunday December 07, 2014 @08:26AM (#8712076)
I would rather guess it is for consistency reasons. People are more familiar with Linux than FreeBSD.

Similarly one of the reasons Facebook relies on PHP is because a lot of developers are familiar with it.

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by justincormack on Sunday December 07, 2014 @11:41AM (#8712671)
I know lots of people who are more familiar with FreeBSD who are working on Linux systems. The Linux monoculture is just massively dominant.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by agumonkey on Sunday December 07, 2014 @07:57AM (#8712024)
Pretty telling technical story and also flattering for the FreeBSD team.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by ps0ps on Sunday December 07, 2014 @11:40AM (#8712666)
This is not true. Facebook has never attempted to migrate them to Linux. If WhatsApp wants to convert to Linux, it is their decision. More than likely they will stay on FreeBSD as that is what they know how to run and a good way to kill a product is to force them to do things instead of focusing on building their product.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by sandGorgon on Sunday December 07, 2014 @12:58PM (#8712935)
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by toast0 on Sunday December 07, 2014 @01:08PM (#8712974)
A job posting in August indicates a failed migration attempt that couldn't have started until October, because that's when the transaction closed?
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by wslh on Sunday December 07, 2014 @08:14AM (#8712054)
> Linux cant cope with that amount of traffic

I like FreeBSD but I don't think we can say that Linux can't cope with that amount of traffic while Google and Facebook use Linux. There is a missing piece here that's not about network traffic.

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by justincormack on Sunday December 07, 2014 @08:21AM (#8712065)
I think it is amount of traffic per server. WhatsApp didnt have a lot of servers and maybe reengineering for massive scale out on Linux doesnt make sense versus staying with what works.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by nonane on Sunday December 07, 2014 @09:40AM (#8712279)
Are there more details available about this? What specifically about the Linux TCP/IP stack is weaker than BSD's stack? Where does the weakness lie?
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by jlouis on Sunday December 07, 2014 @10:01AM (#8712336)
It doesn't have to be a weaker stack per se. It is also a lot about knowledge. WhatsApp vested much time into understanding FreeBSD. It may be if you vested the equal amount of time on Linux with knowledgable people, then you would see roughly the same performance.

If I should take a guess, the advantage of the FreeBSD stack is that it is simpler and easier to understand. Hence, it is easier to tune for performance.

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by andor on Sunday December 07, 2014 @08:25AM (#8712073)
Yes, of course, and Whatsapp users care very much about what's going on behind the scenes ;-) I'm not a mainframe user because I book flights, or a Windows user because i tried Bing.
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by tachion on Sunday December 07, 2014 @10:51AM (#8712489)
It takes very little sense of humor to detect the fact that the title is a nerdy joke ;)
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by sandGorgon on Sunday December 07, 2014 @01:33PM (#8713055)
Correct me if I'm wrong, but one of the biggest scaleups in FreeBSD seemed to be zero-copy sockets. This should be solved in Linux when kdbus is turned on (replacing the current IPC : D-Bus).
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by glasz on Sunday December 07, 2014 @12:38PM (#8712878)
just throwing myself in here: Starve [1] also runs on FreeBSD.

[1] https://www.starve.io

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by sandGorgon on Sunday December 07, 2014 @01:01PM (#8712951)
if one HAD to choose a non-erlang stack (for whatever reason), what would you choose ?

Go, Nodejs or Java - I'm mainly concerned about clustering, because I think that is what gave the Whatsapp devs their ability to scale(without having to do too much re-engineering).

Or can the clustering be moved to something like the queuing layer ?

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by pjmlp on Sunday December 07, 2014 @01:18PM (#8713013)
Java, of course.

There are lots of JVM implementations to choose from, many with high quality code generation (both AOT and JIT) and GC implementations.

It has VM and cluster monitoring tools only comparable by .NET offerings.

The majority of big data solutions are built on top of it.

There are lots of libraries to choose from.

If Java 8, the language, isn't your thing, there are lots of modern languages to choose from.

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by sandGorgon on Sunday December 07, 2014 @01:28PM (#8713038)
I wsa hoping you wouldnt say that :(

Java high performance clustering is... intimidating. It is too enterprisey. From my (little) research, it seems that one would need to build this ground up on top of Netty or Undertow.

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by marktangotango on Sunday December 07, 2014 @02:20PM (#8713230)
Care to elaborate? Load balancing to a bunch of http servers, sticky sessions, or store the session to a db, or the best option is no session data, memcached, its all pretty straight forward?
Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by dschiptsov on Sunday December 07, 2014 @11:39AM (#8712662)
Showing FreeBSD/Erlang stuff that just works and cope with substantial amounts of traffic is not politically correct, because it will cause a severe pain of cognitive dissonance to the Docker/JVM/Node/Cloud crowds..

Lets chant together: JVM is the best VM in the world with millions of man-hours invested in optimizations. Servlet/JSP are mature, battle-tested, enterprise-grade, type-safe solutions. NodeJS is the only way to develop high-scalable internet services. Everything must be a containerized micro-service, orchestrated by Docker/CoreOS/systemd.

Re:Rick Reed – WhatsApp: Half a billion unsuspecting FreeBSD... by pjmlp on Sunday December 07, 2014 @01:36PM (#8713064)
It is worse than that, where you would offshore development like on the other cases? :)