A nit: Erlang is strongly typed. You cannot (ok, excluding numbers) almost transparently convert a string into a number into a tuple like you could in C, to which everything is merely memory so casting allows trivial but potentially erroneous conversions. Erlang is dynamically typed, not statically typed. Dialyzer + type annotations (and some inferred) allows for static analysis, but it's not directly part of the compiler so it can't be strictly enforced (automatically).
EDIT:
Also, consider that erlang processes are a dual of objects in the OO sense. They can receive various messages and respond by changing state and/or transmitting messages. This interaction is similar to the way objects (instances of classes) behave in OO languages. The difference being that they're all running concurrently. So you could put a massive amount of state into one process. OR you could have a handful of processes that act together like you have a handful of classes in an OO language.
If I write a server for playing a game, I don't include all of a player's state and their connection in one process. I have a process that handles the network connection. It sends messages to a process (or processes) that handle player state. Which connect to some game state processes, and on until the network handler sends messages back to the player or gets more messages from the player.
Huge parallelism is easy if your data and processes are largely independent, but the real world is rarely so kind.
But you've also increased your reliabily as well. Who cares if the tight single threaded application with a shared heap cand handle 100K connections, if as soon as one of those connection leads to a segfault, all the other 99999 crash as well.
Your tradeoff is between “tightly bound by shared data structures” vs. “tightly bound by process synchronization”. I don't see how either is better than the other.
> the real world is rarely so kind
In the real world, from what I've seen, while everything is interconnected to everything else, not all the connections are equally strong or important. If you want to compute exact results, without possibility of failure, no matter what the computational cost, then sure, you need to take all the connections into account. If you can trade some accuracy for performance gains in the average case, you'll probably want to find ways to prevent minor failures from bringing down the entire system.
The dual of object types (records of methods) are sum types: Object types are defined by how you can eliminate them (calling a method on an object), whereas sum types are defined by how you can introduce them (applying a constructor to suitable arguments).
In my experience with Elixir, objects, method calls and mutable data are easier to write than processes, messages and immutable data. It's not the mutable and immutable part, it's more about the boiler plate of spawning processes, receiving messages and matching them to dispatch them to the appropriate functions. Ruby's and Python's class and method definitions are much more compact and less error prone. Unfortunately they are not as good at parallelism. I wonder if it could be possible to have an OO language with an automatic Erlang process per object, automatic immutable data (just don't let reassign values to variables, like Erlang), and a way to define supervisor trees.
Is it possible to implement it using the metaprogramming features in Elixir?
That's a pretty big exclusion. There are times one really doesn't want type promotion from a particular number representation for performance or accuracy reasons, and if your dynamic typing system does not allow contracts to reject numerical types that will implicitly promote, you can run into huge performance problems.
An example: suppose you want to divide a list of numerators by a single denominator producing a new list. Each division is not guaranteed to produce an whole number. What's the type of the resulting list? What is the type of each member of the resulting list? What is the type of a sum of the resulting list? Is the per-division cost roughly constant or does it depend on the particular numerator, denominator pair?
Bear in mind not just differences between integer and real numbers, but that real numbers may be exact or inexact, and that there may be various representations of inexact numbers (from native floating point to various types of bigfloat).
Really strict typing -- whether in a dynamically typed language with contracts or similar mostly runtime mechanisms, or in a statically typed language with type checking at compile time -- forces you to deal with these questions explicitly. In this case, you might specify that the source list of divisors is a list of positive integers, and that each division produces a single floating point result (and thus you wind up with a list of floats; and you'd specify that the sum of the resulting list should be a float too). That sacrifices precision for performance, which will tend to matter depending on the content and length of the source list.
There are plenty of dynamically typed languages that get fiddly trying to avoid turning some or all of the operations into much more precise types than single float (or even doing exact representations, rational number style), and the performance impact can be dramatic.
Conversely, you may not want to lose precision as you move away from +-0.0f, so you may want to specify that exact arithmetic will be used in the operations in this example instead.
Erlang doesn't have contracts, but it has pattern matching. You can write a function in Erlang that is guaranteed to return only a list of integers let's say, by either converting them, or not accepting lists of floats in later part of your code. On top of this, you have Dialyzer which can warn you when your code doesn't do this (if you typed your functions correctly).
So it's not new. It doesn't make Lisp any less dynamically typed as a language, and it is wholly optional. Newer dynamic languages also let one do some partial or "gradual" typing where a programmer wants to use it.
There is a cost to this at function calling time, but then there is also a cost if one manually programs in a check on the type of an argument, for instance. Good compilers, however, can prove that functions that aren't of (or exported to) global scope will never be called with anything other than the specified types, and will omit the type-checking code.
Additionally, there is plenty of research into interoperation between code in statically typed languages and dynamically typed ones.
Naturally you can always convert back after your arithmetic operator produces the wrong type. But that can be expensive in itself, and it hurts more when the arithmetic operator could have performed a much cheaper operation.
A way around this of course is to eventually de-polymorphize the potentially-expensive operator and program in a hopefully cheap type-check by hand, in write-generically-first/optimize-(or even make correct)-after fashion.
When I learned a bit more about Erlang, I realized that the point is that there are a lot of applications that are IO bound rather than computation bound, and for those apps Erlang performs very well and is easy to program in. Crunching numbers the fastest isn't always the most important thing.
The other advantage is power of abstraction. With Erlang it is easy to describe and program distributed system with lots concurrent components. Sure you can do it with C++, Java, and Node.js etc, but those things are awkward there -- either threads share memory and have to deal mutexes, or you in are in callback/promise hell.
But Erlang isn't really for doing computation, it's for communication. If you have an existing erlang system that has a lot of data and you want to perform some computations on all of it, you'd probably use erlang to get the data onto the appropriate servers and then spawn another process, that can compute efficiently, with numpy or just C or similar.
Distributed systems aren't just a way of scaling performance beyond the number of CPUs you can have using the same memory. It's about reliability. Erlang's design enables you to create network services that have decades of uptime. It achieves this through its concurrency model, through its error handling approach, by allowing hotswapping code (an operation that's relatively easy to reason about without mutable state), and probably more ways that I as an outsider am not aware of.
[1] http://benchmarksgame.alioth.debian.org/u64q/measurements.ph...
[2] http://benchmarksgame.alioth.debian.org/u64q/measurements.ph...
[3] http://benchmarksgame.alioth.debian.org/u64q/measurements.ph...
What makes scaling up hard is moving data around. Once you have more than a single computer, there is no way you can easily share memory between them, so you have to impose some kind of copying for the system to work. If you want to demux a stream for multiple workers, you have to distribute work to the workers. If you have massive amounts of data in a cluster, you have to move the computation to the nodes in the cluster on which the data resides.
Moving data around requires you to have good orchestration of "mostly stateless" computations, with a couple of pinches of persistence strewn in as well. You can do this well in any language, but what makes Erlang well suited for it is that it provides some decent primitives for you with a lot of time sunk into the architecture. Beating this architecture in any other system requires you to spend some time doing that. And chances are it isn't as general, so when the world around you change, the framework you used is left behind.
Before Erlang, Tandem systems built hardware/software with many of the same ideas in them. They built these systems primarily for fault tolerance and robustness, but they found, somewhat to their surprise, the same architecture is good at scaling. The reason I believe, after 10 years of Erlang programming, is mostly that the computation model of isolated services forces you to think distribution into the system from day one. Your solution naturally gravitates toward the distributed model, and this in turn means it is easier to scale out later. The model also makes it hard to accidentally build a part of the system which can slow down everything. I think this should be given more credit than it is normally given.
And once you have your problem distributed, you call into that CUDA GPU code on the node to obtain the high computation speed. Or you call into your FPGA or DSP ASIC. Any problem on the CPU is slow because of its general purpose behavior (the exception: You are Fabrice Bellard)
Indeed, Jim-gray's (from tandem) paper 'why computers stop and what we can do about it' is an quite good. It contains a detailed report of machine failure including s/w and h/w and details techniques for reducing the mtbf by these.
Erlang's language and runtime seems to have picked seminal ideas from here...
For your convenience: http://www.hpl.hp.com/techreports/tandem/TR-85.7.pdf
Because they are different problems. Concurrency, meet parallelism.
Concurrency: many smaller tasks that can be multiplexed over one core. The core doesn't need to be particularly fast; it just needs to be able to handle multiple tasks in-flight at once. Web serving, etc.
Parallelism: one big, honking task that can be split over multiple cores (or nodes), and each core needs all the horsepower you can eke out, so it can finish its part of the task quicker. Big Data, etc.
Erlang is not particularly fast (bad for parallelism), but it excels at concurrency.
> I know people do some big data stuff with Erlang
Not sure who is doing big data with Erlang, or why anyone would want to -- unless they have some very fundamental misconceptions about the problem at hand and the tools available.
> compromise some of the ideals espoused in this article to make it work
The article was really nonsense. Most of the terms it throws around have nothing to do with Erlang specifically, and proclamations like Erlang being used for a hypothetical "data center on a chip" are... what? The author's gist basically boils down to this: Use supervision trees with unikernels. I have had the same fantasy, for what it's worth.
Parallelism and Concurrency are not mutually exclusive terms. All parallelism is concurrent.
Concurrency is whenever more than one thing is happening at the same time conceptually. Everything from iterators to threads.
Parallelism is whenever more than one thing is happening at the same time physically. From a user-space perspective, this means threads or a coprocessor (like a GPU).
There are many, many ways to think about this difference, and it's fun (and beneficial!) to do so every once in a while.
> From a user-space perspective, this means threads
Interestingly enough, before SMP and multicore, thread-level "parallelism" was actually disguised by a time-sharing concurrent implementation. That remains true for most threading libraries in languages with a GIL, and any time you have threads exceeding the number of physical cores. In fact, the primary purpose of threads was (and still mostly is) to get a semblance of concurrency... Even the Erlang implementation was single-threaded 2008 or so.
It's the fact that they might run at the same time. Also from the perspective of the programmer, there's no difference between two threads running at the same time or by time sharing, since preemptive multitasking is non-deterministic and has the same implications as true parallelism.
> from the perspective of the programmer, there's no difference between two threads running at the same time or by time sharing
is incorrect when you're talking about computational parallelism (as OP was), because you're not going to realize any speedups with time sharing. In that case, you're using threads as a concurrency mechanism -- not for parallelism.
Nokia created an open-source Hadoop-replacement called Disco [0] that used Erlang for coordination/orchestration -- an underappreciated strength of the language -- of map-reduce jobs, where the jobs were written in Python (and later OCaml, etc.). They've shown that it can handily outperform Hadoop (at least in the canonical wordcount example shown in this talk[1] -- there may be other examples, I haven't actually watched the talk yet). They've used it to mine terabytes of logs, daily, as described in this talk[2] and others apparently have used it as well.
From the abstract[3] describing the first talk, about the project:
We will describe our experiences using Erlang within Nokia to build Disco, a lean and flexible MapReduce framework for large-scale data analysis that scales to large clusters and is used in production at Nokia. Disco is an open-source project that was started in 2008 when attempts to use Hadoop to analyze data proved to be a painful experience. The MapReduce step formed only a portion of the analytics stack, and it was felt that it would be faster to write a custom implementation that would integrate well, than adapt Hadoop with the amount of internal Hadoop expertise available. Among the crucial tasks of such an implementation would be to deal with cluster monitoring, fault- tolerance, and the management and scheduling of a large number of concurrent and distributed jobs. To keep the implementation simple, the use of a platform that provided first-class support for distribution and concurrency was imperative. This motivated the choice of Erlang/OTP to implement the core control plane of Disco. It bears stressing that this choice was driven primarily by pragmatic concerns, as opposed to any beliefs about the superiority of functional programming languages in general or Erlang in particular.
The project's homepage [0] has information, a link to its Github, etc.
[1] https://youtu.be/IjOGUC-iR_Q
[3] http://cufp.org/2011/disco-using-erlang-implement-mapreduce-...
Riak?
This assumes a pedantic qualification of "big data" handling -- does storage count, as opposed to the actual processing of the data? But I think that's an important distinction in this context, as it strikes the heart of the dichotomy between concurrency and parallelism.
(Moreover, I've never really been sure how much Riak is being used for actual 'big data', as opposed to being a master-less, highly available repository for 'regular data' (distinction between 'big' and 'regular' deliberately left vague). More so since Riak is key-value as opposed to columnar, but I suppose that depends on your workload. But that's all besides the point here.)
I am also an Erlang noob, but I don't think pattern matching has to account for all parameters. To my understanding erlang pattern matching is very efficient, and anything you would pattern match on function parameters, you would have to do some type of logic in the function if you had no pattern matching, so I don't see how pattern matching would negatively effect performance.
ETS tables to share data between threads actually make a lot of sense if you think of your program with the idea that no matter what "x" is, this process should properly handle it, because you can't guarantee when the message you sent will be processed.
An ETS table is a DB, but you can replicate it by having a single genserver hold that data, and use calls to retrieve and manipulate it. It gives you one spot for your data to be. If you had that same data in messages or function call stacks, by the time the process executes it, it could be stale.
Its useful if you need one "true source" of individual pieces of data that needs to be synchronized. Its about decoupling data from processes.
It would be interest to re-engineer a language today that tries to fit into Erlang's niche but has a stronger performance focus. Rust, Cloud Haskell, and Go all sort of cluster in the area I'm thinking about, but none are quite what I'm thinking of. Cloud Haskell is probably closest but writing high-performing Haskell can be harder than you'd like. Rust shows you don't have to clone Erlang's immutability and all the associated issues it brings with it for inter-process safety, but Rust of course is "just" a standard programming language next to what Erlang brings for multi-node communication.
In the aughts Ruby and Python were really slow so if you had computational-heavy problems you had to drop into C.
It worked but C isn't great. The thing is - we have a lot of languages that can do computational problems easily now - Rust, Go, Nim, and the list goes on....
It becomes relatively trivial to create libraries which could wrap these languages for Erlang/Elixir. In the few cases where Elixir isn't fast enough, just drop down to something else. Write a small piece of code in Rust (you may as well call Elixir/Rust peanut butter and jelly). Optionally, you could create some kind of DSL which compiles down to another language in Elixir. I did the same with xjs (Elixir syntax, Javascript semantics) [0], and I must say, for a 200-line hack it works really well.
This isn't even considering the fact that a lot more work could be invested in Erlang's VM. Yes, Ericsson is its corporate sponsor but imagine if you had companies trying to make it fast the way they try with Ruby, Python, JS, or another of other more complicated languages.
I think this is relatively low-hanging fruit. You can't tell me that Erlang and Elixir are harder to make fast than other dynamic languages (in most contexts).
And yes, I am doing this, and the personal reason is for fault-tolerance and the professional reason (or how I get time to do it) is based on security criteria.
Speaks the Erlang network protocol and impersonates
an Erlang node on the network. Fully capable of
bi-directional communication with Erlang.
You can also use c nodes (or the JVM interface, which is pretty similar to c nodes I think).
... You can also use ports which define an interface for communicating with external processes.
The world is your oyster!
A language that allows mutable state aside from at process scope is also a no-go. In Erlang you're not supposed to think about what code will be running on which machines as you write the business logic, so you have to assume that every process is running on a seperate machine with no shared memory, so no mutable state above the process level. And mutable state at local scope makes hot swapping code messier, although that's easier to work with than static typing.
Nothing about Erlang is inherently slow. Someone could hire a bunch of developers from v8 or Spidermonkey or maybe just Mike Pall to write a better Erlang runtime.
"A language that allows mutable state aside from at process scope is also a no-go."
Well, I did say the language I'm spec'ing probably doesn't exist. Rust is an interesting example of what can be done to make it so that not actually copying memory is safe, but you'd still have to do some work to make it do the copying more transparently across nodes, which is why I said it's "just" a regular language from Erlang's point of view.
"Nothing about Erlang is inherently slow."
I now believe that dynamically-typed languages that are not built from speed from the beginning (LuaJIT being pretty much the only reason I even have to add that parenthetical) are inherently slow. I've been hearing people claim for 20 years that "languages aren't slow, only implementations are", I've even echoed this myself in my younger days, yet (almost) none of the dynamic languages go faster than "an order of magnitude slower than C with a huge memory penalty" even today, after a ton of effort has been poured into them. Some of them still clock in in the 15-20x slower range. Erlang is a great deal simpler than most of them, and I don't know whether that would net an advantage (fewer things to have to constantly dynamically check, although "code reloading" mitigates against some of these) or disadvantage (less information for the JIT to work with). Still, at this point, if someone's going to claim that Erlang could go C speed or even close to it in the general case, I'm very firmly in the "show me and I'll believe it" camp.
At some point it's time to just accept the inevitable and admit that, yes, languages can be slow. If there is a hypothetical JS or Python or PHP interpreter that could run on a modern computer and be "C-fast", humans do not seem to be capable of producing it on a useful time scale.
Heh it is funny. I might have an idea why. It is hard to use regularly to perform day to day releases. Simply because building correct apups and so on take a lot of time. Most systems are already decomposed into separate nodes and cand handle single node maintenance, so that is what we do at least. Take a node down, upgrade, bring it back up. Care has to to taken to have mixed version in a cluster but that is easier than proper 100% clean hot upgrade.
But having said that, I have used hotpatching by hand probably 5-6 times in the last couple of months. Once on a 12 node live cluster. That was to fix an one-off bug, for that customer before having to wait for a full release, another time was to catch an function_clause error that was crashing gen_server and so on. It was is very valuable having that ability.
> till, at this point, if someone's going to claim that Erlang could go C speed or even close to it in the general case, I'm very firmly in the "show me and I'll believe it" camp.
It doesn't matter if it goes C speed, it has the fault tolerance, expressive language, it is battle tested, it has good runtime inspection and monitoring capability, if someone came one day and said you lose all those but you gain C speed, I wouldn't make that trade.
One is where you live upgrade a full running release, including all applications and version, where you mutate state that had its format changed. All this in production, without any downtime. This is incredibly hard to get right. Erlang gives you a lot of tools (OTP & friends) to achieve this, but it is still very complex.
The other is reloading Erlang code in a runtime system. I.e. recompiling and reloading one or several modules in a runtime system. This is usually done during development (see Phoenix for Elixir for example) or perhaps even in production when you know what you're doing. This is relatively easy, with some risks of course if you are doing it in production.
An example would be changing log level settings. Normally Elixir log blocks can be compiled entirely out when running in production mode. But it should be possible to fairly safely recompile with debug logs enabled and reload without missing a beat.
---
There are a number of other dynamically typed languages that have fast implementations. Javascript has a few; Common Lisp, Self, Julia, to name some others. They'll never be as fast as C, certainly not when comparing highly optimized programs, but they're fast. It looks like most dynamically typed languages can be made to run 10x slower than C. Compare that to CPython and HiPE which are more like 100x slower.
I don't think code reloading would hurt JIT performance too much. The prerequisites for runtime specialization of procedures basically accomodate eveything hot swapping would need. I also think the way people use Erlang's type system is probably more ammenable to conservative type inference than the existing fast dynamic languages, and that's one of the more important metrics.
Interesting. What are the exceptions you have in mind to warrant the (almost) hypothetical? You mention LuaJIT. I've also heard that Q/KDB are quite fast. Anything else?
One language that I think is woefully underappreciated for how ubiquitous it is is GLSL. With OpenGL 4 Compute Shader you get surprisingly close to general-purpose use for the type of tasks that benefit from massive parallelism. And GLSL is really quite a nice language; driver bugs are the main things holding it back.
Maybe for highly SIMD stuff but it seems like pull in a bunch of baggage around GPUs, etc.
On the other hand it would force you to do your data partitioning right up-front(much like the SPUs on the PS3).
Then again I'm used to mobile GPUs where any conditional statement used to cause the shader to be evaluated 2^n for each and the gathered at the end(aka forget about any branching).
For my 2c I'm a fan of Elixir + Rust, Rust has a nice C ABI that should make it easy to embed.
Every single state change requires a function call, every function call involves a pattern matching algorithm that has to account for all of your program arguments (effectively almost the entire state of your thread!) and isn't even strongly typed. And then there is synchronization and data sharing between threads, which seems a bit handwavy and effectively requires a database running in your program that the threads can poll.
If your problem set is lots and lots of small independent tasks that don't have to finish overly quickly, then Erlang is fantastic. Stuff like switching voice circuits for example. But I'm trying to imagine manipulating a 1TB dataset with thousands of worker threads if you have to make a copy on the stack for every change every worker makes.
I know people do some big data stuff with Erlang, so these have to be solved problems somehow, but I can't help but to suspect that they have to compromise some of the ideals espoused in this article to make it work.