News for hackers. Stuff that matters.
Re:Managing two million web servers by akkartik on Sunday March 13, 2016 @09:08PM (#11280354)
This article got me to go figure out precisely what Erlang processes are. Tl;dr - they aren't OS processes. So it is still conceivable that an error in Erlang can bring down all your web servers.

http://stackoverflow.com/questions/2708033/technically-why-a...

Re:Managing two million web servers by toast0 on Monday March 14, 2016 @01:53AM (#11281288)
An error in the beam VM (or a nif) could lead to crash of the OS process, and related termination of all of the erlang processes within. However, the beam VM is pretty robust, because of the design, it turns out to be quite small and relatively simple[1], so you don't tend to end up with things were weird edge cases cause terrible crashes that are hard to find. For nifs you write yourself, it's up to you :)

Process isolation isn't complete though, one erlang process can use all of many of the limited resources and end up with the whole vm shutting down. A single process can't use all the CPU, but it can use all the memory, all the atoms, all the filedescriptors, all the ets tables, etc; basically everything in the system limits[2]

[1] Calling it simple is a compliment -- a normal person can look at the vm code and understand it, if they need to.

[2] http://erlang.org/doc/efficiency_guide/advanced.html#id70200

Re:Managing two million web servers by akkartik on Monday March 14, 2016 @02:11AM (#11281332)
Thanks, yours was finally the comment I was hoping for.

Could you point me at the right place to read (or read about) the beam sources? I'm looking at https://github.com/erlang/otp/wiki/Routemap-source-tree -- is there something better?

Re:Managing two million web servers by toast0 on Monday March 14, 2016 @12:40PM (#11283821)
You're in the right place; unfortunately there's not a lot of good documentation on the internals (although do check out erts/emulator/internal_doc ). It's much easier to dive into the source if you have something specific on your mind (like debugging a performance problem, or if you find an beam crash...). erts/emulator/beam has the most basic parts of the virtual machine -- processes, memory allocation, process switching, message passing, interfaces to the operating system (although a lot of that is in erts/emulator/drivers), etc.
Re:Managing two million web servers by rdtsc on Monday March 14, 2016 @02:30AM (#11281375)
> So it is still conceivable that an error in Erlang can bring down all your web servers.

No it won't. Not if you mean an error in an Erlang process. It almost sounds impossible, but that is the beauty of the BEAM virtual machine. It is really a marvel of engineering.

Now if you mean an error in Erlang VM itself, then yeah, if that crashes it will bring down the all the connections. But even that there is a answer -- distributed Erlang. Erlang VM can talk to processes on other Erlang VMs, even if those are running on another server on other side of the planet.

Re:Managing two million web servers by signa11 on Monday March 14, 2016 @12:08AM (#11281002)
> ... So it is still conceivable that an error in Erlang...

are you talking about erlang-runtime here ? it is not very clear what you are implying. can you please elaborate ?

if i take your statement to mean 'erlang runtime' then this is no different than saying, if there is a bug in glibc which gets tickled when i do something, then my process will crash, or if there is a bug in jvm, then the application hosted on the jvm will crash etc. etc.

Re:Managing two million web servers by akkartik on Monday March 14, 2016 @12:21AM (#11281050)
From OP: "In Erlang we create very lightweight processes, one per connection and within that process spin up a web server. So we might end up with a few million web-servers with one user each."

I don't see the difference between spawning a thread for each connection in other languages and spawning a process for each connection in Erlang, so I was curious if the word 'process' implied some extra level of insulation. Reading all the comments here it's still not clear to me what benefit is being obtained by having "2 million web servers" rather than 2 million connections.

Re:Managing two million web servers by phamilton on Monday March 14, 2016 @12:30AM (#11281079)
Shared memory space is one thing. A process has isolated memory. An error in a spawned thread could mess up shared memory, crashing multiple threads. An error in a process can't mess with the memory space of other processes.

Since there's no shared memory, performance characteristics change. Garbage collection is done per process and so "stop the world" doesn't happen, meaning each individual "web server" isn't at the mercy of other requests for performance.

It's lots of little things like that.

Re:Managing two million web servers by rdtsc on Monday March 14, 2016 @02:34AM (#11281381)
> I don't see the difference between spawning a thread for each connection in other languages and spawning a process for each connection in Erlang,

That is a huge difference. Otherwise by now, why would Erlang not just run pthread calls and just spawn a thread. Why bother doing all that work?

The reason is because Erlang processes do not share heap memory. That is one of the most important features of Erlang runtime. A crashing process can just crash on its own without leaving all the other 1999999 processes in an undetermined state.

Re:Managing two million web servers by true_religion on Monday March 14, 2016 @03:31AM (#11281499)
It also simplifies garbage collection. Rather than having to stop the world for garbage collection, it happens on a per-process basis.

Additionally, if a process terminates before a garbage collection cycle is run the VM just reclaims all of its memory without any additional computation needed.

Re:Managing two million web servers by tvon on Sunday March 13, 2016 @11:41PM (#11280912)
I don't follow, how does Erlang processes not being OS processes relate to what could happen if an Erlang error occurred?
Re:Managing two million web servers by ghayes on Sunday March 13, 2016 @09:12PM (#11280379)
Well, not if you distribute your erlang processes across several physical nodes.
Re:Managing two million web servers by jbardnz on Sunday March 13, 2016 @09:39PM (#11280510)
You mean just like you can with Apache or Nginx?
Re:Managing two million web servers by statictype on Sunday March 13, 2016 @11:23PM (#11280854)
Yes - it's exactly like that. The Erlang VM itself has to be rock solid just like you expect Apache and Nginx to be rock solid.

Fortunately this more or less seems to be the case.

Re:Managing two million web servers by phamilton on Sunday March 13, 2016 @11:53PM (#11280957)
I would say a better comparison is the Linux kernel. If the kernel crashes, everything on that box goes.
Re:Managing two million web servers by felixgallo on Sunday March 13, 2016 @10:15PM (#11280642)
No. Not at all like those.
Re:Managing two million web servers by niij on Sunday March 13, 2016 @11:03PM (#11280807)
Can you please expand on why?
Re:Managing two million web servers by vertex-four on Monday March 14, 2016 @06:24AM (#11281963)
The processes can communicate with processes on other servers in exactly the same way that they can communicate with processes on the same server.

If you have one traditional application server and want to do some form of cross-user interaction - let's say chat - you can do that trivially, put it in a queue for that user in a global map. Now when you outgrow that server, you need to rewrite all your code to understand the concept of users being on other servers or use an external message queueing system.

In Erlang, all of this is built in by default - if you write code for the OTP framework (the standard library for dealing with messaging, process supervision, etc), all you need to do is connect the two servers together and point them at the same shared user->process mapping process (which you have to build whether you're dealing with one server or 20, as there's no global data otherwise).

Of course, if you have absolutely no direct interaction between users, it's trivial to scale anything - fire up a new server and direct some portion of your traffic at it. Erlang's trick is to make it that easy even when you do have direct interaction between users. And of course that works for even backend workloads - if you have a backend server that your frontend servers talk to and need to scale it, if you've coded in Erlang and put as much logic as possible in per-connection processes, you're probably a significant chunk of the way there.

Re:Managing two million web servers by ww520 on Monday March 14, 2016 @03:59PM (#11285134)
What does any of these have to do with the equivalence between Apache/Ngix and Erlang when running multiple servers of them to avoid crashing all?
Re:Managing two million web servers by vertex-four on Monday March 14, 2016 @04:20PM (#11285287)
You can't use traditional servers to scale in lots of cases without a lot of additional development work. You can use Erlang servers to do so. Therefore, they're not the same - Erlang covers a broader range of use cases.
Re:Managing two million web servers by reinhardt1053 on Monday March 14, 2016 @06:34AM (#11282010)
A constructive comment would be appreciated.
Re:Managing two million web servers by ww520 on Sunday March 13, 2016 @10:45PM (#11280757)
The Erlang error in code could be replicated to every single physical node and all of them would crash.
Re:Managing two million web servers by rusanu on Monday March 14, 2016 @01:47AM (#11281277)
Presumably one would catch such a blatant error early in testing. The problem is the odd case, the code that only crashes in a rare state, never seen in test. When the odd state does occur the effect is mild, as only one actor dies as opposed to the entire OS process.
Re:Managing two million web servers by ww520 on Monday March 14, 2016 @03:55PM (#11285105)
I was responding to the assertion that distributing code to multiple "processes" would avoid crashing all of them, which is simply not true. Even the rare state you mentioned would crash all the processes. The rare state would be an input that testing failed to catch. The code in any of the processes encountering the rare input could crash.
Re:Managing two million web servers by felixgallo on Sunday March 13, 2016 @10:50PM (#11280774)
That is not what happens.