News for hackers. Stuff that matters.
Re:Elixir and Phoenix after two years by blunte on Monday April 05, 2021 @04:08PM (#26703975)
I should have been clearer in my final note. My Elixir vs Python example was for standalone batch (script) processing. It was not in the context of a webapp.

The case was a recurring calculation system which would take a few hundred thousand records and do about a dozen different calculations against those (some calculations requiring additional lookups).

The original script was in Python, and then after the new webapp was built in Elixir, I rewrote it also in Elixir. I discovered that even tuning the batch sizes, the Elixir/Ecto version was much slower than the Python version. Then I looking into per-thread Elixir performance and found out that that is not its strong suit :). So the answer there of course is to leverage the strength of the tools and parallelize it; but I had no need to do that since I already had a working Python version. Could Elixir have been faster if multi-threaded? Probably. Single? No, I doubt it.

Re:Elixir and Phoenix after two years by ch4s3 on Monday April 05, 2021 @04:48PM (#26704512)
Ahh, ok. You've sort of hit a weird spot for Elixir. Python will just start up faster and that will be noticeable. Math is much better optimized in Python, though this is changing with NX.

The real gain here would probably be streaming chunks of rows out of ecto and maybe firing those off to tasks per CPU core. Which is easier than it sounds.

That said, you had working code so no real reason to rewrite it except as a learning exercise.

Re:Elixir and Phoenix after two years by toast0 on Monday April 05, 2021 @10:36PM (#26707382)
Typical ways to start beam end up doing a lot of work you might not need.

Taking some time to fiddle with the startup settings for a short running task can make a big difference. You might actually get better results by setting it to single cpu, because there's less setup that way. Really depends on how much total cpu time your work takes. If your VM is going to run for a year (or more), startup time hardly matters and so the defaults aren't highly tuned for a 30 second script.

Re:Elixir and Phoenix after two years by dnautics on Monday April 05, 2021 @04:11PM (#26704024)
Hm, if you have a chance to revisit it, you may want to consider Flow, which will likely take a lot of the pain of parallelization away.