One possibility is you're using a single process instead of parallelizing things. For example, you may want to use one process per event, etc. Though if the hardware is very underpowered and say single core, I could see it becoming problematic.
Unfortunately.
From metrics, computing AWS signatures takes up an absurdly large amount of CPU time. The actual processing of events is quite minimal and honestly well-architected, a lot of stuff is loaded into memory rather than read from disk. There's syncing that happens fairly frequently from the internet which refreshes the cache.
The big problem is each event computes a new signature to send back to the API. I do have to wonder if the AWS signature is 99% of the problem and once I take that burden off, the entire system will roar to life. That's what makes me so confused because I had heard Erlang / Elixir could do on the scale of significantly more per minute even with pretty puny hardware.
One thing I am working on is batching then I am considering dropping the AWS signatures in favor of short-lived tokens since either way, it's game over if someone gets onto the system anyway since they could exploit the privilege. The systems are air-gapped anyway so the risk is minimal in my opinion.
Oh, sounds nice! Caching in Elixir really is nice.
Okay, that makes sense. Elixir isn't fast at pure compute. It can actually be slower than Python or Ruby. However, the signatures likely are NIFs (native code). If the AWS signs are computed using NIFs then the CPUs are likely just can't keep up with them. Tokens would make sense in that scenario. But you should check the lib or code you're using for them.
> The big problem is each event computes a new signature to send back to the API. I do have to wonder if the AWS signature is 99% of the problem and once I take that burden off, the entire system will roar to life. That's what makes me so confused because I had heard Erlang / Elixir could do on the scale of significantly more per minute even with pretty puny hardware.
Yeah, crypto compute can be expensive especially on older / smaller cpus without builtin primitives. Usually I find Elixir performs better than equivalent NodeJS, Python, etc due to it's built in parallelism.
Also one thing to lookout for would be NIF C functions blocking the BEAM VM. The VM can now do "dirty nifs", but if they're not used and the code assumes the AWS signs will run fast, it could create knock on effects by blocking the Beam VM's schedulers. That's also not always easy to find with Beams built in tools.
On that note, make sure you've tried the `:observe` tooling. It's fantastic.
> One thing I am working on is batching then I am considering dropping the AWS signatures in favor of short-lived tokens since either way, it's game over if someone gets onto the system anyway since they could exploit the privilege. The systems are air-gapped anyway so the risk is minimal in my opinion.
Definitely, seems logical to me.
I'll have to rig up Observer. I've been using recon because I was being lazy overall.
https://github.com/ExWeb3/ex_keccak
for my usage, benchmarks went from 2.79K ips to 346.97K ips
1: https://hexdocs.pm/broadway/Broadway.html
> I also looked into the wake-up lengths of threads. I am starting to get my head around "dirty schedulers" but I am not entirely sure how to affect those or how I can besides it doing it forever me.
Note that dirty schedulers really only affect NIFs which run longer than what the BEAM schedulers expect. I mentioned that in regards the possibility that the AWS sigs are taking longer than they should, then they'd cause havoc on the scheduler.
I'm also suspicious of the `+scl true` setting as maybe being a bit of a red herring. I've been using BEAM off and on for 10 years both professionally and as a hobbyist and I've never used this myself nor seen anyone else ever need to use it. I'm sure there are circumstances where someone, somewhere has used this, but in a given Elixir app it is extremely likely that there is lower-hanging fruit than messing with scheduler flags.
In terms of queuing, are you using Oban or Broadway or something hand-built? It's common for folks to DIY this kind DAG/queuing stuff when 99.9% of the time using something like Oban or Broadway would be better than hand-rolling it.
I actually think Elixir really doesn't have great performance. TechEmpower, which is IMO the most real world standardized tests out there shows that Phoenix doesn't even complete. And Elixir+Plug+Ecto performs worse than Rails, which is an entire framework.
Everyone in Elixir land tells me "Oh those benchmarks don't matter". Yet they are heavily talked about, and referred to here and other places. They only don't matter if you perform terribly on them I suppose.
And they say "Oh we didn't care to put much effort into it", Yet Jose Valim himself tried to work on it and didn't fix it. He's written extensively about how this type of test doesn't really fit elixir, etc, but ultimately it's just doing DB queries, why does this not work?
I really think Elixir is mostly propaganda at this point. It's a huge mental paradigm shift and I have seen myself that it wasn't performant, and as you said you keep thinking "Oh I must be doing it wrong".
I just cannot fathom why anyone in a decent sized company would use with all the negatives it has going for it. YMMV
Proof below
https://www.techempower.com/benchmarks/#section=data-r22&hw=...
This is inaccurate. I have started looking again into solutions only last week [1]. My suspicion was always the database pool size was too small but, when I tried to contribute 4+ years ago, fine tuning was hard because it took too long, so I didn't pursue it further [2].
My discontent with the benchmarks is that they are not measuring what people effectively run in production. Since you mentioned Rails, here is how a Rails application looks like:
https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
But almost nobody runs a Rails application like that in production.
And then you look at the configuration of most stacks and they have been explicitly fine tuned to match exactly the concurrency traffic and patterns that the benchmark sends through. But, for most web applications, your web traffic is not homogeneous and you don't have a predetermined number of concurrent requests.
I still believe those benchmarks are not indicative of what you will actually see in production. Most companies who have gone from Rails to Phoenix, for example, report a 10-20x reduction in operation size and costs. But it is clear at this point people put way too much stock on these benchmarks. The irony of it all is that, if someone copies these setups into their actual applications, they will most likely perform worse. Oh well.
[1]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/9302
[2]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5432 - here you can see me increasing the database pool to 40... but most benchmarks today run with 512-1024 connections (which, once again, is most likely not what you would do in prod). In any case, we need to bump our numbers accordingly.
The database pool needs to be at 512 to 1024? Yet Rails seems to get away with far less than that and still have better performance.[2]
Even if, somehow, Elixir is actually faster, which seemingly lacks evidence at this point other than some anecdotes by people who have a vested interest in people adopting the language (primarily Elixir dev shops and framework/language authors), the fact that the community of Elixir devs cannot get decent results on these tests says a lot.
If these people can't get good results in tuning Elixir, who can?
Not a surprise to see people ITT also noting that they ran into the same types of problems that the tests reveal.
[1]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
[2]: https://www.techempower.com/benchmarks/#section=data-r21&hw=...
Rails concurrency model is per process. So you will have `n_workers` pools and the amount of connections will be `n_workers * 3` based on the code you shared. The production machine has 28 total threads [1], which means Rails is starting 35 workers, according to their auto tuning code [2]. Overall, Rails uses 35x more pools than the Phoenix application and has twice the number of connections (105). It is absurd to imply that Rails is actually running with far less.
Second of all, the database pool needs to be 512 or 1024 if your app server can push traffic quickly enough to the database. If you are increasing the Rails pool size and it is not getting faster, then Rails is your bottleneck. Based on the runs I did last week, it seems the pool is the bottleneck in Elixir's case. We will know for sure in a couple weeks once the PRs land.
> If these people can't get good results in tuning Elixir, who can?
You can look at the results from the latest community efforts in main: https://www.techempower.com/benchmarks/#section=test&runid=1...
If you want to compare with Rails in particular, we beat Rails in every benchmark except two:
1. Single query: which will be tackled by better sizing our pools
2. Plain text: where we are currently sending the wrong payload (but when we sent the proper payload, as in the last published round, we beat Rails too)
And it does not include any of my proposed changes yet.
My point is exactly that the feedback cycle is long, so it takes time, and I was not able to invest time before. But given how intent some folks are to misinterpret benchmarks (and assign malice to the whole community), it is probably better to continue looking into it.
[1]: https://ark.intel.com/content/www/us/en/ark/products/120474/...
[2]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
I wrote, while learning, a monitoring system in Elixir. I put everything in one "Phoenix" app. What does it do?
* Query about 60 oracle databases in parallel every five seconds.
* Query a number of KDB databases every five seconds.
* SSH to dozens of machines every ten seconds
* Ingest statistics from icinga.
* Pull ticket updates from Jira every couple of seconds.
* Provide facilities for connecting in to screen on about 100+ remote machines, and interacting with their interfaces.
* API access to internal and external commands/functions.
* more.
It does much more than this, while serving live-view sessions to users, on a dual core machine (CPU: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz x 2), while averaging about 8% CPU for everything it does. The SSH component replaces a python version that does the same thing. The Python instance, by itself, pushes a 6 core box (same xeon as above but with 6 cores, rather than 2) to 100% CPU, and spaws close to 3,000 OS tasks.
Many of the other checks that are running in that 8% CPU Elixir process replace python, node, and java checks, each of which required more ram, demanded more CPU, and were spread across eight to ten dedicated machines per check. We have been retiring machines, and reducing CPU load, so I'd say we do see the 0 to 20 reduction.
Now, my solution might have been written better, but that was what I came up with while learning Elixir.
Although I still strongly disagree with the initial observation that they are "real world". Most applications in the real world receive heterogeneous traffic, from a varying and unknown number of clients, with logging, potential contention on resources, and other aspects that are not taken into consideration.
I am working on an Elixir project that in no circumstance should be taxing but it's falling over miserably at like 100 - 200 events per minute. The detail is it is distributed-ish IoT. I didn't write it, the person(s) who did write it are gone, and no one else writes Elixir. I've gotten some good gains already but I'd like to squeeze the 80 - 90% of juice I think I can get before resorting to beefier hardware.
I've gotten into instrumenting and measuring it and I have some ideas but I'd love to hear others point me to other ideas. The real problem is that the hardware is miserably underpowered and it is real-time, by that I mean I can't defer, schedule for later, or de-prioritize anything.
---
To actually contribute, I really like Elixir. I am not yet sure why I would advocate for it over something more 'simple' like nodejs (My background is, accidentally, Javascript World) but it's certainly a very nice language to write in. It feels magical but not too magical where you get scared it's trapping you into its web.
Before anyone jumps too much on me for it, I gauge "simplicity" by how many people can I hire to write it. You can barely swing a cat without hitting 3 competent Javascript developers. I tried for many years to hire another golang dev so I could write it professionally, I only encountered a few despite having been in most interviews my employers would do. With that said, it may just be that the Venn diagram between "writes Javascript" and "writes golang" is small.