afhammad

Good opportunity to share some competitive resilience: https://monzo.com/blog/tolerating-full-cloud-outages-with-mo...

Could you share more on your local setup please?

Similar but for the London Undergroound:

https://traintimes.org.uk/map/tube/ https://traintimes.org.uk/map/tube/schematic/

There are a few mentions of Oban [1] here. Most people don't realise that Oban in fact uses SKIP LOCKED [2] as well.

Oban's been great, especially if you pay for Web UI and Pro for the extra features [3]

The main issue we've noticed though is that due to its simple fetching mechanism using locks, jobs aren't distributed evenly across your workers due to the greedy `SELECT...LIMIT X` [2]

If you have long running and/or resource intensive jobs, this can be problematic. Lets say you have 3 workers with a local limit of 10 per node. If there are only 10 jobs in the queue, the first node to fetch available jobs will grab and lock all 10, with the other 2 nodes sitting idle.

[1] https://github.com/sorentwo/oban [2] https://github.com/sorentwo/oban/blob/main/lib/oban/engines/... [3] https://getoban.pro/#feature-comparison

"chasing" is a good word

From my understanding, `work_mem` is the maximum available memory per operation and not just per connection. If you have a stored procedure with loops and/or many nested operations, that can quickly get quite big.

One trick worth noting, is that you can override the working memory at the transaction level. If you have a query you know needs more memory (e.g doing a distinct or plain sorting on a large table), within a transaction you can do:

`set local work_mem = '50MB'`

That will override the setting for operations inside this transaction only.

> Bad code in a specific part of the codebase bringing down the whole app, as in our November incident.

This is a non-issue if you're using a Elixir/Erlang monolith given its fault tolerant nature.

The noisy neighbour issue (resource hogging) is still something you need to manage though. If you use something like Oban[1] (for background job queues and cron jobs), you can set both local and global limits. Local being a single node, and global across the cluster.

Operating in a shared cluster (vs split workload deployments) give you the benefit of being much more efficient with your hardware. I've heard many stories of massive infra savings due to moving to an Elixir/Erlang system.

1. https://github.com/sorentwo/oban

Good timing, I’ve been searching for something like this!

tax exemption

413 Payload Too Large

There's been a lot of development in the Elixir ecosystem and so a lot to share.

Embedded with Nerves, the various AI/ML libraries and tooling, LiveBook getting better every day, LiveView innovating and improving DevEx, etc..

Then there's fly.io pushing out content given their target market is developers and they have a big focus on deploying Elixir applications.

Adding my vote here. Throughout my career I've gone the C#.NET -> RoR -> Clojure -> Elixir/Phoenix route. After spending a year with Elixir, iterating rapidly on a production system, I'd default to it for any MVP or serious project going forward.

LiveView is also very close to becoming default over React for interactive Frontends.

It seems that this wasn't as routine as these things aught to be but rarely are. Could this have been an ad hoc test in response to potential escalations from Russia?

Retool (on-prem) + Tailscale (golink, etc)

A lot of people are in the right place at the right time, but very few are equipped to realise it and take advantage of it.

At the beach, waves will hit everyone in their path (right place, right time) but only those that notice it and already have momentum in the direction it's heading are able to ride it. If they have experience, and equipment (a board), they can ride it for longer without crashing.

By Louis Pasteur. It's quoted in the article

On the tools: I have gone the route of using the libraries (e.g Kaffy for Phoenix, ActiveAdmin for Rails) and later building a central shared backoffice from scratch that any team can contribute to. In the former you are always wasting time fighting the library to get what you want, in the latter it has the advantage of being language/framework agnostic but it's too costly for smaller teams.

More recently i've used tools such as Retool (there are many alternatives), it has been ideal and allows for non engineers to build there own tools with a little bit of SQL knowledge.

Fair, I was going based on hearsay/docs. It looks like LiveView has been trying to accommodate the issue with some work arounds such as: https://github.com/phoenixframework/phoenix_live_view/issues...

Alpine.js is seeing an increase in utility as it pairs nicely with htmx[0], Phoenix LiveView[1] and the likes for lightweight interaction that doesn't require the server.

[0]: https://htmx.org

[1]: https://github.com/phoenixframework/phoenix_live_view

why? (actually curious)

> almost a WebUI alternative to a REPL

That's basically how my team have been increasingly using it. Simply connect Livebook to a locally running Phoenix project and you have a Livebook REPL into your server. When you're dealing with complex data, pulling from different sources and have to build up a bunch of context before you iterate on a function it's super useful to be able to break up that code into chunks, take form inputs[0] along the way and document any quirks. We keep a bunch of livebooks committed in the repo to help debug and iterate on the more complex parts of our codebase.

0: https://hexdocs.pm/kino/Kino.html

> Maybe a more serious question: is anyone using Elixir ML in production?

Checkout this recent talk about how https://www.amplified.ai/ moved from Python to an Elixir ML stack: https://www.youtube.com/watch?v=Y2Nr4dNu6hI

You can't generalise, and nobody really knows.

Yep agreed. I think/hope an incident such as this will create a step change in their security processes. As you said, it only takes a few of these incidents to make it very difficult for the business to recover, especially given the tech savvy customer base.