Not to refute your experience, but "adding true -> ok to every if statement" sounds like an anti-pattern. A couple of notes to elaborate:

1. If-statementes should rarely be used in Erlang because case is preferred

2. If you're returning ok everywhere, you could also just let it crash by doing something akin to true = function(). Always go for let it crash first, unless you really know you can handle the error sensibly (and then use a case-statement)

3. httpc is kind of known to not be the most formidable HTTP client. Hackney is a more modern and better choice. The good thing with httpc is that it is included with OTP, otherwise there are better choices (scalability- and feature-wise)

4. When someone says "multicore support" and "performance" support is not great it's usually one of two things: (a) they're developing a use case that is not fit for Erlang (e.g. compute heavy) or (b) they don't know how to use Erlang properly (e.g. too complex process setups)

Now, many of these things are not obvious to people new to Erlang and takes some experience to know about. This I would say is the bigger problem with the Erlang eco-system.

Agreed. Couple of clarifications/observations.

(1) there are situations like this:

    if
    DeductCreditInRatingGroup ->
      log("deducting credit...");
    end;

unfortunately you cannot write it this way. You have to write it as:

    if
    DeductCreditInRatingGroup ->
      log("deducting credit...");
    true -> ok
    end;

Not a huge deal but it's annoying.

(2) Let it crash attitude is something I never understood. It's just not something you can do most of the time. If subscriber consumed 2MB and you let it crash in the middle of rating function you just "lost" 2MB. With couple of million subscribers on LTE that turns into very expensive error handling very fast.

(3) We actually ended up using hackney.

(4) We were not doing anything compute heavy. I think our application was well architected and we had no problem fitting it into OTP "framework". Comparing CPU and memory usage to our Java implementation, Java is more performant.

edit: fixed code formatting.

1. Here I would do something like this:

  log("deducting credit...", DeductCreditInRatingGroup),

  ...

  log(Msg, true) -> do_the_logging();
  log(Msg, false) -> ok.

Nowadays though, you should really use the new logger introduced in Erlang 21. There you can do run-time filtering on many different parameters, or even write your own handlers. Using if-statements is usually considered an anti-pattern in Erlang, and I think very common coming from languages where only if-statements exist. Problems are almost always better solved by using case, or even better, pattern matching in function heads.

2. Sounds like one of the rare cases where you care about the error and can also handle it. In the use case you mentioned, I would buffer the data outside of the loop that produces it and wrap the loop in a try-catch statement. That way, you don't loose the data if one iteration crashes.

4. Erlang usually ends up being good at latency and concurrency (scalability over cores), together with a smaller and easier to read code base. I'm curious about your case, and what would have made Java a better fit. If you share a lot of global objects, Java might be faster for some things (at the cost of concurrency usually). Erlang has tools for that as well though (e.g. ETS and recently persistent_term).

(1) was not about logging. I was trying to illustrate a valid use case for `if` statement that requires `true` guard expression. We used lager[1] for logging. I understand that new logger was inspired or loosely based on lager.

(2) I would argue that it's a rare case you don't care about errors.

(4) Multicore support in Erlang was not always great and you would run into all kinds of problems after 2-4 cores. Than for a while it was 12-16 cores. Not sure what the limit is these days.

Again, I am not arguing Java over Erlang. There is no lack of great examples of large successful project running on Erlang. For me and my team Erlang was just too much hassle for not much gain. Your milage may vary.

[1]https://github.com/erlang-lager/lager

Can't comment on your other issues (I'll say I have noticed the complete opposite in practice, where the same team working with Java vs Erlang led to far more reliable, performant systems in Erlang, but I obviously can't speak to your experience), but for (2) -

You always care about exceptions/errors. The thing is, most you won't predict/can't handle. Those you do predict, and know how to handle, you should, because they're not really errors/exceptions at that point; they're just edge cases.

The point of Erlang is not to make it so you just throw instead of addressing an edge case; it's to make it so you reason about how, if something you don't predict and don't know how to handle happens, you can get back into a known good state. It's actually phenomenal at doing that. I've had complicated user facing production systems work without noticeable issue for years (even while under active development) in Erlang. I've never seen that in any other language. Not to say it can't happen, I just haven't seen it, and the operational lift to achieve that was no different for us than just "make it work"; we were spending the same amount of time thinking about failure cases with the Erlang system as other languages. The difference was a better approach to handling the things we didn't think of.

1. My point still stands, using if-statements is not common in Erlang exactly for the reason you mention.

2. In Erlang it usually only make sense to care about errors which you can actually handle in a sensible way. Everything else you usually "let crash" and let the supervisor tree deal with recovering. Defensive programming is another anti-pattern in Erlang in my opinion.

4. Not sure what you mean with "not great". With Erlang, if you have a completely parallelizable problem you should see close to linear scalability. If it drops of after that it's either because the problem is not parallelizable enough or because of your architecture.

Yeah, I'm not arguing Erlang over Java either. I'm just pointing out that blanket statements like this are usually because of architecture or design rather than shortcomings of the language itself.

1) I would just write

    DeductCreditInRatingGroup andalso log("blah")

2) Let it crash attitude is often misinterpreted. It means your system should be fault-tolerant in general, so you can focus on the happy case instead of trying to predict all humanly possible failure modes. If you _expect_ invalid input, then handle it properly.

> Let it crash attitude is something I never understood. It's just not something you can do most of the time. If subscriber consumed 2MB and you let it crash in the middle of rating function you just "lost" 2MB. With couple of million subscribers on LTE that turns into very expensive error handling very fast.

Let it crash is the only sensible thing to do. Depending on the type of error, the 2MB could be absolute garbage. I would rather stop sending any message instead of sending potential wrong data without knowing.

I can't refute your personal experience, but 10 years of Erlang work left me longing for more. It sounds as if perhaps Erlang was not used optimally, or, alternatively, best practices may not have been followed.

10 years ago the ecosystem was horrendous, and is much better now. Granted, the build experience could be better but I didn't find it that bad.

I find that people's experiences are relative to what they are accustomed to. Early years for me included compiling and linking C in Windows (segmented) and that was truly horrible!

It's funny how "you're holding it wrong" is the go to response whenever somebody criticizes Erlang. Why is that?

Replace "Erlang" with any other language and the statement still holds true.

If you want the obvious explanation, it's because we all have different experiences, and no one tool will be the biggest win for all teams across all domains.

I mean, what you said about if statements (which should be very very rare in Erlang) kind of betrays it: you most likely _are_ holding it wrong.

Probably because they really are holding it wrong?

did you use Vert.x ? IMHO that has the closest cognitive match to Erlang.

it regularly tops performance benchmarks - https://www.techempower.com/benchmarks/#section=data-r18&hw=... and super simple heroku deployments - https://github.com/vert-x3/vertx-examples/tree/master/heroku...

Also Kotlin !! https://vertx.io/docs/vertx-core/kotlin/ (kotlin vertx on heroku: https://github.com/vert-x3/vertx-examples/tree/master/kotlin...)

Interesting, do you think Elixir would be better in your case? Also what was the amount of requests per second? Looks like you didn't really need Erlang's scalability.

No; as I understand it Elixir's only advantage is, arguably, better syntax. I don't mind Erlang's syntax that much.

We were handling, in peek hours, over 100k requests per second spread over 3 servers.

Funny you should mention scalability as something that somehow justifies Erlang's other shortcomings. I don't think scalability is something Erlang/OTP does out of the bag. One thing we learned the hard way is that Erlang does not do overload well at all. Erlang mailboxes are unbounded and it can get very messy very fast. I always found that design choice odd given Erlang's origins in telecommunications.

Now this is not to say you can't do massively scalable systems in Erlang, it's just that there's no silver bullet when it comes to scalability.

The point about Elixir is it does better in ecosystem, libraries, tooling and removes a lot of complexities you named.

Every time I wade into Erlang projects, I’m struck by the cultural and overall ecosystem differences.

Scalability isn't something any language does out of the box. The design and implementation of the application determines whether it scales or not. In that sense, Erlang scales about as well as any other language. It does have some features that might make scalability easier to implement compared to some other languages, but you have to know how to use it properly.

Exactly my point. It scales about as well as any other language so why not use a language that's nicer to work with?

> It scales about as well as any other language

You can get just about anything to "scale" given architectural compensations (like scads of containers), so this, ipso facto, doesn't mean much.

> so why not use a language that's nicer to work with?

Well, why not indeed? But here are some thoughts about that.

Nothing is a panacea, Erlang included. Some of the ecosystem is ugly and hard to work with, but improving. Erlang is in a vicious cycle:

* Erlang ecosystem sucks

* Fewer people want to use it

* Fewer people are available to improve the ecosystem

* Fewer people are available to fill jobs for Erlang

* Less demand for Erlang

* Loop

Erlang gives you certain things that no other language I know of does, and these things make it a more robust, easier to program, highly concurrent solution without needing architectural compensations for those things.

That doesn't mean that you can magically throw 1 million TPS at Erlang running on N boxes and your internal queues/mailboxes won't overflow. You, along with the rest of the world, need backpressure handling mechanisms.

What Erlang gives you that Java doesn't is totally decoupled processes that cannot directly affect each other. No mutexes or semaphores, no shared global memory.

It also gives you ultra-lightweight processes that garbage-collect independently of each other, that are cheap enough to start and stop that you can create one per incoming connection and make it completely unaffected by any trouble that any other process gets into. This is not true of threads in Java.

It gives you a supervisory framework that, if used judiciously (yeah, yeah, no true Scotsman, but that doesn't make this any less valid) gives you an ultra-robust system that degrades gracefully under error or overload conditions.

It gives you linked processes that detect a broken process and brings down the others (if you so choose) to avoid leaving orphaned processes cluttering up the system, plus automatic restart, baked in. These do come free of charge when using OTP supervisory frameworks in the recommended fashion.

I think that in some cases, Erlang has been overhyped and people come into it expecting it to magically solve all the hard problems. People that overhype Erlang do it a disservice. For example, the stupid "nine nines" thing was an almost once-off, limited situation in a very constrained environment and I am sure many Erlangists wish it had never been mentioned.

And I wish nobody had ever said the words "let it crash", because they have been misunderstood and taken out of context and again done Erlang a disservice. Just like the phrase "premature optimization is the root of all evil" has caused untold damage when it was (and is often) taken out of context.

The whole "let it crash" thing was shorthand for "if a process cannot reasonably handle and error condition itself, let it die and be reincarnated by the supervisory process". It does not mean "just let everything crash". For example, if you have a system that has long-lived connections, maybe (say) http/2 connections, or XMPP, or Apple push, you most definitely do not want to let it crash without doing everything possible to recover from errors such that the connection stays up under as many error conditions as possible.

But for the most part, a process should not be written defensively and try to recover from errors where it has no business doing so - it should leave it to the supervisor.

I have written Erlang systems, heavily used in production that ran reliably, without crashing, for many months. Usually they were only stopped to do an OS upgrade. They weren't error-free - we found out about some persistent errors by checking the logs occasionally, then fix them and often hot-patch the fix into the production systems.

Again, Erlang is not the world's best programming language or ecosystem. It is, within its design envelope, one of the finest soft real-time distributed multiprocessing environments around when used, like anything, with skill and careful architectural design.

In my humble opinion.

I call that "handling back pressure", because I don't know what to call it.

I don't know of any system, stack that handles it directly, natively. Though my sample size is pretty small.

> We were handling, in peek hours, over 100k requests per second spread over 3 servers.

How big was one server and what does one request entail?

What Java framework (for concurrency) do you use?

If you use Java I'm assuming you are not using process. If so, I believe Phoenix is no worse than Java applications in terms of ease of use and scalability etc.

And it in terms of performance it indeed require some design experience on Erlang process. But it shouldn't be bad. After all WhatsApp and Discord have run on top of it with very few engineers to support massive customers before. I doubt online charging system could be more chatty.

> Surprisingly enough multicore support was not great

Considering that it’s state of the art, you must have been doing it terribly wrong.

> adding `true -> ok` to every if statement

That confirms it.

That's just not true. Before you would run into all kinds of bottlenecks after 2-4 cores. I believe it was addressed in R13A[1].

[1] http://erlang.org/download/otp_src_R13A.readme

Not after 2-4 cores. And R13A is 2009, that was 10 years ago. (Java 8 is 2014.)

Today’s Erlang (R22) MP support is practically the best, bar none.

Hence the past tense.