News for hackers. Stuff that matters.
Re:Ask HN: Why Isn't Elixir More Popular? by josevalim on Thursday October 10, 2024 @02:02AM (#41795989)
> And they say "Oh we didn't care to put much effort into it", Yet Jose Valim himself tried to work on it and didn't fix it.

This is inaccurate. I have started looking again into solutions only last week [1]. My suspicion was always the database pool size was too small but, when I tried to contribute 4+ years ago, fine tuning was hard because it took too long, so I didn't pursue it further [2].

My discontent with the benchmarks is that they are not measuring what people effectively run in production. Since you mentioned Rails, here is how a Rails application looks like:

https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

But almost nobody runs a Rails application like that in production.

And then you look at the configuration of most stacks and they have been explicitly fine tuned to match exactly the concurrency traffic and patterns that the benchmark sends through. But, for most web applications, your web traffic is not homogeneous and you don't have a predetermined number of concurrent requests.

I still believe those benchmarks are not indicative of what you will actually see in production. Most companies who have gone from Rails to Phoenix, for example, report a 10-20x reduction in operation size and costs. But it is clear at this point people put way too much stock on these benchmarks. The irony of it all is that, if someone copies these setups into their actual applications, they will most likely perform worse. Oh well.

[1]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/9302

[2]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5432 - here you can see me increasing the database pool to 40... but most benchmarks today run with 512-1024 connections (which, once again, is most likely not what you would do in prod). In any case, we need to bump our numbers accordingly.

Re:Ask HN: Why Isn't Elixir More Popular? by DoesntMatter22 on Thursday October 10, 2024 @09:06AM (#41798402)
Sounds like you are agreeing with me. You tried to work on it years ago and didn't fix it. Even when it was completing last year it's performance was half of what Rails was, which isn't known to be fast [1].

The database pool needs to be at 512 to 1024? Yet Rails seems to get away with far less than that and still have better performance.[2]

Even if, somehow, Elixir is actually faster, which seemingly lacks evidence at this point other than some anecdotes by people who have a vested interest in people adopting the language (primarily Elixir dev shops and framework/language authors), the fact that the community of Elixir devs cannot get decent results on these tests says a lot.

If these people can't get good results in tuning Elixir, who can?

Not a surprise to see people ITT also noting that they ran into the same types of problems that the tests reveal.

[1]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

[2]: https://www.techempower.com/benchmarks/#section=data-r21&hw=...

Re:Ask HN: Why Isn't Elixir More Popular? by josevalim on Thursday October 10, 2024 @10:04AM (#41798952)
> The database pool needs to be at 512 to 1024? Yet Rails seems to get away with far less than that and still have better performance.[2]

Rails concurrency model is per process. So you will have `n_workers` pools and the amount of connections will be `n_workers * 3` based on the code you shared. The production machine has 28 total threads [1], which means Rails is starting 35 workers, according to their auto tuning code [2]. Overall, Rails uses 35x more pools than the Phoenix application and has twice the number of connections (105). It is absurd to imply that Rails is actually running with far less.

Second of all, the database pool needs to be 512 or 1024 if your app server can push traffic quickly enough to the database. If you are increasing the Rails pool size and it is not getting faster, then Rails is your bottleneck. Based on the runs I did last week, it seems the pool is the bottleneck in Elixir's case. We will know for sure in a couple weeks once the PRs land.

> If these people can't get good results in tuning Elixir, who can?

You can look at the results from the latest community efforts in main: https://www.techempower.com/benchmarks/#section=test&runid=1...

If you want to compare with Rails in particular, we beat Rails in every benchmark except two:

1. Single query: which will be tackled by better sizing our pools

2. Plain text: where we are currently sending the wrong payload (but when we sent the proper payload, as in the last published round, we beat Rails too)

And it does not include any of my proposed changes yet.

My point is exactly that the feedback cycle is long, so it takes time, and I was not able to invest time before. But given how intent some folks are to misinterpret benchmarks (and assign malice to the whole community), it is probably better to continue looking into it.

[1]: https://ark.intel.com/content/www/us/en/ark/products/120474/...

[2]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

Re:Ask HN: Why Isn't Elixir More Popular? by DoesntMatter22 on Thursday October 10, 2024 @12:37PM (#41800611)
Nice to see elixir put up decent benchmarks for once. It looks like it's about 80 percent faster. Still a far cry from "10 to 20x reduction in size and cost".
Re:Ask HN: Why Isn't Elixir More Popular? by josevalim on Thursday October 10, 2024 @12:51PM (#41800769)
YMMV. In benchmarks like cached queries, it already outputs a solid 5.5x faster than Rails, and it is worth remembering that the Rails app is way too stripped compared to what folks are actually running in prod (which was one of my original criticisms).
Re:Ask HN: Why Isn't Elixir More Popular? by bc58 on Friday October 11, 2024 @06:54PM (#41814674)
10 to 20 improvement here.

I wrote, while learning, a monitoring system in Elixir. I put everything in one "Phoenix" app. What does it do?

* Query about 60 oracle databases in parallel every five seconds.

* Query a number of KDB databases every five seconds.

* SSH to dozens of machines every ten seconds

* Ingest statistics from icinga.

* Pull ticket updates from Jira every couple of seconds.

* Provide facilities for connecting in to screen on about 100+ remote machines, and interacting with their interfaces.

* API access to internal and external commands/functions.

* more.

It does much more than this, while serving live-view sessions to users, on a dual core machine (CPU: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz x 2), while averaging about 8% CPU for everything it does. The SSH component replaces a python version that does the same thing. The Python instance, by itself, pushes a 6 core box (same xeon as above but with 6 cores, rather than 2) to 100% CPU, and spaws close to 3,000 OS tasks.

Many of the other checks that are running in that 8% CPU Elixir process replace python, node, and java checks, each of which required more ram, demanded more CPU, and were spread across eight to ten dedicated machines per check. We have been retiring machines, and reducing CPU load, so I'd say we do see the 0 to 20 reduction.

Now, my solution might have been written better, but that was what I came up with while learning Elixir.

Re:Ask HN: Why Isn't Elixir More Popular? by DoesntMatter22 on Friday October 11, 2024 @07:40PM (#41815018)
That seems like a good fit for Elixir tbh
Re:Ask HN: Why Isn't Elixir More Popular? by gregors on Thursday October 10, 2024 @12:54PM (#41800795)
The techempower repo is literally a gift to the world. I've seen and learned many ways of optimizing code ...almost none of which should ever be used in production or would ever make it past a PR at most companies. Still, it's amazing in the amount of non-realistic approaches used. It really is a work of art.
Re:Ask HN: Why Isn't Elixir More Popular? by DoesntMatter22 on Friday October 11, 2024 @01:02AM (#41806334)
I hate when people downplay TechEmpower. They are the best benchmarks going, and awesome because anyone can contribute and as you said there is a lot of learning that can be done there.
Re:Ask HN: Why Isn't Elixir More Popular? by josevalim on Saturday October 12, 2024 @04:21AM (#41817422)
You are right. There are many positive things about TechEmpower and keeping it open to all and running takes an incredible amount of effort, which is definitely appreciated and I failed to recognize. That was unfair of me.

Although I still strongly disagree with the initial observation that they are "real world". Most applications in the real world receive heterogeneous traffic, from a varying and unknown number of clients, with logging, potential contention on resources, and other aspects that are not taken into consideration.