Our CI builds our elixir artifacts, updates them to GCS, then our minions pull the artifacts, untar them and move them into the right place. Aside from deploying go binaries, Elixir binaries have been the next easiest thing.
We have one (very low throughput) service that runs Elixir inside of a docker container, however, the rest just run BEAM on the VM directly, with no containerization. BEAM at higher load does not enjoy sharing cores with anyone. Most of our Elixir services sit at 80-100% of CPU on all cores. BEAM likes to spin for work, and is CPU topology aware. Two BEAM scheduler threads running on the same core is a recipe for disaster. Since we're fully utilizing our VMs, and shipping the tars is simple enough, Docker gave us too much operational overhead for it to be worth it.
We've also had our fair share of troubles running the BEAM VM on GCP at scale. I could go into this in more detail if anyone's curious.
Please do!
- CPU utilization can differ wildly on Haswell & Skylake. On Skylake processors our CPU utilization jump by 20-30% due to Skylake using more cycles to spin. Luckily, all of that CPU time was spent in spinning, and our actual "scheduler utilization" metric remained roughly the same (actually, on Skylake it was lower!).
- Default allocator settings can call malloc/mmap a lot, and is sensitive to latency on those calls. Under host memory bandwidth pressure, BEAM can grind to a halt. Tuning BEAM's allocator settings is imperative to avoid this. Namely, MHlmbcs, MHsbct, MHsmbcs and MMscs. This was especially noticeable after meltdown was patched.
- Live host migrations and BEAM sometimes are not friends. Two weeks ago, we discovered a defect in GCP live migrations that would cause a 80-90% performance degradation on one of our services during source-brownout migration phase.
GCP Support/Engineering has been excellent in helping us with these issues and taking our reports seriously.
I thought that GCP live migrations were completely transparent for the kernel and the processes running in the VM. I'd be happy to read a bit more about the defect that made BEAM unhappy.
Excessive allocations and memory bandwidth are two very different things. Often then don't overlap, because to max out memory bandwidth you have to write a fairly optimized program.
Also, are the allocations because of BEAM or is it because what you are running allocates a lot of memory?
We were able to simulate this failure condition synthetically by inducing memory bandwidth pressure on the guest VM.
We noticed that during certain periods of time, not caused by any workload we ran, the time spent doing malloc/mmap would 10-20x, but the # of calls would not.
I'm curious what tools you used to discover this
Docker allows me to run various Phoenix/Elixir apps with various versions of the language and framework since I don't have time to update all of them. My method also allows me to compile .tar.gz releases on my local machine since the lower-tier servers that I can afford don't always have enough memory to compile there.
Here's a tutorial I wrote on how I deploy my Phoenix apps if anyone is interested: https://gist.github.com/jswny/83e03537830b0d997924e8f1965d88....
In particular since the final product is on-prem, the highly available, let-it-fail philosophy is appealing to reduce the number of issue tickets we're likely to have to deal with.
Don't take my strategy as gospel - I am on an open-ended, highly divergent branch of our company's product line and have been given a lot of product leeway by management that is.... "open-minded".
https://github.com/cogini/nerves_system_ec2
I'm excited to play with it in the future.
It works pretty well, and I get to ship artifacts rather than source code.
There's a PaaS called Gigalixir that has a lot of promise, but you'll be limited to one app at a time, and it must be re-deployed at least every 30 days or it gets shut down.
Also been using Heroku for some Elixir apps in production.
Also a simple Dockerfile to push to AWS
Thanks!