News for hackers. Stuff that matters.
Re:Enigma: Erlang VM Implementation in Rust by hopia on Thursday April 30, 2020 @11:16AM (#23031745)
A new Erlang VM with just replicated functionality is a fairly hard sell to the Erlang/Elixir community, who brag with the industrial track record of the BEAM.

I believe you'd get much more interest if there was some ambitious new promise for this new VM, such as 10x sequential performance etc.

Re:Enigma: Erlang VM Implementation in Rust by dnautics on Thursday April 30, 2020 @12:25PM (#23032676)
Someday this is going to need to happen though. IMO, "the right way" to do this is via the strangler pattern:

https://www.michielrook.nl/2016/11/strangler-pattern-practic...

Probably the language that is most poised to achieve this is Zig; it would be feasible to start by wrapping the entire BEAM in a zig compilation unit; which at the very least potentially offers an easier path to maintaining the codebase across multiple platforms. Followed by hodgepodge doing bits and pieces in zig, which could be achieved via straightforward transliteration at first.

The very different mindset of the rust PL lends itself to total rewrites, which I don't think will sit well in the BEAM community. On the other hand erlang has tons of strange rewrites happening over its own internal ecosystem all the time (gen_fsm -> gen_statem, pg -> pg2 -> pg), etc.

Re:Enigma: Erlang VM Implementation in Rust by muizelaar on Thursday April 30, 2020 @12:59PM (#23033130)
Do you know of any examples of this being done with Zig? I can think of a couple with Rust:

- https://gitlab.gnome.org/GNOME/librsvg completed a migration to Rust.

- https://github.com/RazrFalcon/rustybuzz and https://github.com/immunant/rexpat are making decent progress.

Re:Enigma: Erlang VM Implementation in Rust by dnautics on Thursday April 30, 2020 @01:58PM (#23033870)
No, because zig is still in 0.6.0, and the BDFL says "don't use this in prod yet"? Yeesh.
Re:Enigma: Erlang VM Implementation in Rust by catach on Thursday April 30, 2020 @09:27PM (#23038223)
Perhaps then, for the sake of setting those expectations, the wording "the language that may be most poised to achieve this in the future is Zig" would be preferred?
Re:Enigma: Erlang VM Implementation in Rust by dnautics on Thursday April 30, 2020 @10:30PM (#23038533)
Last I checked "poised to achieve this" implies the future. Moreover I have specific reasons to say that, namely that zig now ships with C compilation with transparent cross-compilation support so you can use it as a direct replacement for C and (I believe make), so the statement is based on current zig features whose importance I alluded to in the comment.

Fwiw this is also a real problem as I am often hearing complaints about windows ABI support in elixir, anyways. Moreover, having written nifs, I have to say the c header for nifs is basically unintelligible without manually parsing the DEFINES because of windows cross-support needs.

Please let me know if I'm using "poised" incorrectly.

Re:Enigma: Erlang VM Implementation in Rust by catach on Friday May 01, 2020 @12:35AM (#23039173)
To my mind, "poised" in this context is roughly synonymous with "ready". If the official position of the project is "don't use it yet" then it's not ready yet.

The future implied is the completion of the action one is ready for, not further work of preparation for that action.

Your take may, of course, differ.

Re:Enigma: Erlang VM Implementation in Rust by archseer on Thursday April 30, 2020 @11:54AM (#23032230)
Yeah I definitely didn't see it being production ready any time soon, but I thought it was an interesting project for people that wanted to learn BEAM internals. That's how I got started with it at least, I had problems trying to contribute BEAM just because of the sheer size of the codebase and lacking the domain specific knowledge.

I do think that having alternative implementations is good for experimentation though, similar to how Ruby was improved upon ideas from JRuby and Rubinius, even if most users never used those two directly.

Re:Enigma: Erlang VM Implementation in Rust by bsder on Thursday April 30, 2020 @09:31PM (#23038242)
> That's how I got started with it at least, I had problems trying to contribute BEAM just because of the sheer size of the codebase and lacking the domain specific knowledge.

And the fact that a LOT of BEAM was terribly undocumented. I'm impressed you got anywhere, personally. The last time I looked at BEAM to understand some weird behavior (that eventually did turn out to be a problem in Erlang/BEAM), it was completely impenetrable.

Your implementation is probably useful as documentation, if nothing else.

Re:Enigma: Erlang VM Implementation in Rust by bitwalker on Friday May 01, 2020 @12:19PM (#23043644)
100%, when I started building Lumen, I spent an enormous amount of time working out how various parts of the BEAM runtime were implemented, and it was (and still is sometimes) a grind. That macro-heavy C code is just such a bear to read. I understand why it was written that way, but working with it is just unpleasant.

Having Enigma, and other implementations like it, provides a huge value in terms of understanding how it all fits together - understanding the Enigma implementation and then going and trying to make sense of the BEAM would probably be a way better path than trying to dive into the BEAM straight away.

Re:Enigma: Erlang VM Implementation in Rust by fortran77 on Thursday April 30, 2020 @01:27PM (#23033447)
We do a lot of Erlang work here. The BEAM is so reliable that I wouldn't spend 1 minute looking at an experimental alternative.
Re:Enigma: Erlang VM Implementation in Rust by carapace on Thursday April 30, 2020 @02:37PM (#23034423)
Yeah, this. I'm just getting started with Erlang and I already feel like an idiot for not using it sooner. When I think of some of the things I've done to try to achieve what the BEAM does out-of-the-box...
Re:Enigma: Erlang VM Implementation in Rust by rhlsthrm on Thursday April 30, 2020 @02:49PM (#23034579)
Can you give some examples? I've been getting more and more interested in Erlang.
Re:Enigma: Erlang VM Implementation in Rust by battery_cowboy on Thursday April 30, 2020 @05:18PM (#23036253)
RPC is basically built in, so you'll probably never use REST internally. There's an in memory database built in (ETS) that will replace redis for most key value storage cases. There's easy recovery from crashes via supervision trees and associated features. You can do hot upgrades while your system is fully operational.
Re:Enigma: Erlang VM Implementation in Rust by dnautics on Thursday April 30, 2020 @10:58PM (#23038659)
A few protips: hot reloads are outside of a few corner cases generally unnecessary these days. Be careful about erlang rpc if you expose it in any fashion over the network and follow the eef security working group guidelines re: erlang term format. I use Elixir's plug.crypto.non_executable_binary_to_term/2
Re:Enigma: Erlang VM Implementation in Rust by battery_cowboy on Friday May 01, 2020 @12:31AM (#23039152)
Thanks, I knew both of those bits of info, but I didn't know about the guidelines and how to deal with the RPC issue, very helpful.
Re:Enigma: Erlang VM Implementation in Rust by dnautics on Friday May 01, 2020 @01:50AM (#23039457)
to be fair, the guidelines are hot off the presses. I literally went to CodeBEAM, ran into to Bram Verburg (whose fantastic X509 library i had just found a week prior) and he told me about those guidelines, which had just hit hours before we spoke. Bram is great, he's very modest, and was amused that I thought he was some sort of superstar.
Re:Enigma: Erlang VM Implementation in Rust by battery_cowboy on Friday May 01, 2020 @05:31AM (#23040446)
Well I'm "hot off the press" when it comes to elixir, so it's a good time for me to learn this stuff, thanks for the link below, too.
Re:Enigma: Erlang VM Implementation in Rust by carapace on Friday May 01, 2020 @02:27AM (#23039578)
You wouldn't happen to have a link handy would you? It would be much appreciated.
Re:Enigma: Erlang VM Implementation in Rust by dnautics on Friday May 01, 2020 @03:32AM (#23039869)
Re:Enigma: Erlang VM Implementation in Rust by carapace on Friday May 01, 2020 @11:35AM (#23043073)
Bless you! "You're a scholar and a gentleman." :-D
Re:Enigma: Erlang VM Implementation in Rust by dnautics on Friday May 01, 2020 @07:07PM (#23047422)
Actually I'm a bit of an asshole if you really get to know me, but I'm usually nice on the internet. Thank you!
Re:Enigma: Erlang VM Implementation in Rust by mleonhard on Friday May 01, 2020 @11:19PM (#23048853)
When I used Erlang in 2005, there was a problem with message queues growing and OOM-ing the process. The standard solution to that is RPC flow control and backpressure. Does this problem exist with modern Erlang?

Google's gRPC/Stubby has the same problem. It's one reason why gRPC servers are never deployed on the public Internet and always use hefty reverse proxies.

Re:Enigma: Erlang VM Implementation in Rust by jrobn on Thursday April 30, 2020 @11:05PM (#23038695)
I always feel like Erlang and more generally BEAM is the language for a lot of today’s problems but will never see any kind of traction because of “worse is better”.
Re:Enigma: Erlang VM Implementation in Rust by spockz on Thursday April 30, 2020 @03:06PM (#23034799)
Can you share something on this reliability? Is it more reliable than the JVM? Or is it more predictable in terms of performance? I’ve found the jvm itself to be rock solid as well.
Re:Enigma: Erlang VM Implementation in Rust by hopia on Thursday April 30, 2020 @04:38PM (#23035801)
Erlang VM is consistent/predictable in terms of latency, it's engineered that way. For pure computational performance you won't find it optimal.
Re:Enigma: Erlang VM Implementation in Rust by battery_cowboy on Thursday April 30, 2020 @08:39PM (#23037975)
Erlang/OTP and BEAM have been around a long time and is probably just as mature as the JVM and Java, but they have slightly different goals: very low tail latency, easy error recovery, and easy distributed compute in erlang vs. better customization and median performance for the JVM. I'm sure there's more differences and more nuance than that, but I'd be very happy to use either language/VM in production.
Re:Enigma: Erlang VM Implementation in Rust by dnautics on Thursday April 30, 2020 @11:05PM (#23038693)
It's also harder to write abjectly wrong code in the BEAM. I found out that my code was effectively self-forkbombing, because I had hooked up an error reporter (I adapted myself because the version I wanted had bitrotted) to rollbar which (very rarely, when the service was in outage) cause a crash, which would try to report, crash, and exponentially grow. Amazingly, this caused no service degradation for our customers.
Re:Enigma: Erlang VM Implementation in Rust by conradfr on Thursday April 30, 2020 @01:32PM (#23033524)
There is some go projects like [1] that can connect to Erlang nodes and claim to be speedier.

[1] https://github.com/halturin/ergo

Re:Enigma: Erlang VM Implementation in Rust by themgt on Thursday April 30, 2020 @11:44AM (#23032103)
If the VM is in Rust could it be compiled to WASM?
Re:Enigma: Erlang VM Implementation in Rust by rkangel on Thursday April 30, 2020 @11:52AM (#23032196)
You wouldn't want to, for various reasons. See this blog post about Lumen and the decision decisions:

https://tylerscript.dev/bringing-the-beam-to-webassembly-wit...

Re:Enigma: Erlang VM Implementation in Rust by hobofan on Thursday April 30, 2020 @11:59AM (#23032302)
You wouldn't want to _right now_. However for almost all points there is a solution underway/in planning. In a year or two it might be feasible.

There would however be other limitations, like filesystem APIs etc. not being available in the browser that a lot of frameworks in BEAM languages expect, that would severely limit the usefulness, though I guess that applies to either implementation strategy.

Re:Enigma: Erlang VM Implementation in Rust by rkangel on Friday May 01, 2020 @04:59AM (#23040289)
I can't imagine a situation in which it would be acceptable to download an entire (WASM compiled) copy of the BEAM from every new website I visit.

Full AOT compilation with dead code elimination seems like a much better fit for minimising that bundle size.

Re:Enigma: Erlang VM Implementation in Rust by derefr on Thursday May 07, 2020 @02:46PM (#23106625)
WASM isn't just for the web, though. It's beginning to look a lot like the next "portable ISA" (successor to Pascal's pcode, applet-era JVM bytecode, Google's PNaCl, etc.)

Add a progressive web-app to your mobile's home screen—get a cached WASM bundle, for near-native performance of the resulting "app." Download an Electron app—get a WASM bundle wrapped in a projector.

Even then, there are still browser use-cases where "full fat" apps are no problem. The "apps" in the Chrome Web Store—Adobe has a version of Photoshop in there, I think—make a lot of sense to be WASM. You'll launch them once and keep them open for hours.

Re:Enigma: Erlang VM Implementation in Rust by dnautics on Thursday April 30, 2020 @02:02PM (#23033923)
I think the point though is that architecturally there are performance hits if you don't respect the fact that WASM has a different architecture than what the BEAM expects (harvard vs von neumann IIRC), so you may NEVER want to if you get it right in the first place.
Re:Enigma: Erlang VM Implementation in Rust by wahern on Thursday April 30, 2020 @03:52PM (#23035318)
The Harvard-Von Neumann dichotomy is wrong. C also works perfectly fine on Harvard architectures--it's why function pointers in C are special, aren't guaranteed to be convertible to a void pointer, and why uintptr_t is optional. POSIX adds these additional guarantees to support dlsym, which returns function addresses as void pointers.

The problems with compiling other languages to Web Assembly are primarily 1) lack of goto and 2) inability to instantiate and jump between different stacks. These limitations are especially problematic for languages like Erlang/BEAM and Go because Web Assembly-based VM implementations require an extra level of indirection in order to implement some of their core language semantics, resulting in quite slow performance compared to even a pure, strictly compliant C implementation (and presuming the WASM VM itself adds no overheard, which is not actually the case).

WASM excluded goto support because it was argued that the relooper algorithm required to translate goto constructs to structured WASM statements was sufficiently capable to cover the vast majority of existing code. And they provided evidence to back up that claim. The flaw in that reasoning is that language implementations and similar niche cases have special needs that application code rarely requires, and in that space constructs like goto are crucial to both simplicity of implementation and performance; the inadequacies of relooper become the norm rather than the exception.

Re:Enigma: Erlang VM Implementation in Rust by dnautics on Thursday April 30, 2020 @04:11PM (#23035532)
thanks for the clarification!
Re:Enigma: Erlang VM Implementation in Rust by jlg23 on Thursday April 30, 2020 @06:06PM (#23036758)
Just being able to amend job requirements with "or rust experience" is most probably worth it.