News for hackers. Stuff that matters.
Re:Distributed SQLite with Elixir by Groxx on Wednesday February 15, 2023 @11:10PM (#34814897)
Re:Distributed SQLite with Elixir by mattfrommars on Wednesday February 15, 2023 @11:40PM (#34815104)
> This is because we are not replicating the data between the instances. This is a problem Elixir is built for…

What feature is this and does it not really exist in C# or Java?

Re:Distributed SQLite with Elixir by silbermm on Wednesday February 15, 2023 @11:49PM (#34815173)
The feature is distributed message passing (basically an Actor model) https://www.erlang.org/doc/reference_manual/distributed.html

You can get this in other languages with the help of middleware like RabbitMQ or Kafka, but not many languages have it built in.

Re:Distributed SQLite with Elixir by nesarkvechnep on Thursday February 16, 2023 @02:35AM (#34816050)
You can’t get it in C# or Java. The BEAM virtual machine is build upon the idea of lightweight processes. You might get an imitation in other mainstream places but not the real thing. Take a look at the Actor Model if you haven’t heard of it.
Re:Distributed SQLite with Elixir by mattfrommars on Thursday February 16, 2023 @08:54AM (#34818584)
> might get an imitation in other mainstream places but not the real thing.

Could you further elaborate on this?

I am familiar with Actor Model. I had work related to Akka in Scala that works on the principles of Actor Model.

Re:Distributed SQLite with Elixir by weatherlight on Friday February 17, 2023 @11:04AM (#34836269)
Erlang uses a preemptive scheduler for process scheduling, allowing for parallelization on multiple CPUs, while Scala distinguishes between thread-based and event-based actors. Thread-based actors can perform long-running computations without hindering other actors, but use more memory and scheduling overhead. Event-based actors are more lightweight but not suitable for parallelism as they execute on the same thread.
Re:Distributed SQLite with Elixir by nl on Wednesday February 15, 2023 @11:58PM (#34815236)
This is such a weird post.

They build a distributed counter that is consistent across a cluster using the Elixir message passing thing, but there's this whole thing about using the Fly Distributed SQLite thing as well.

But if they wanted to build a distributed counter they could choose to use Postgres at the first step and then it will just work.

Re:Distributed SQLite with Elixir by Joel_Mckay on Thursday February 16, 2023 @12:21AM (#34815364)
For elixir+phoenix a cockroachDB store abstraction is natively supported.

I like the design in many ways, but still need to find something noncritical to test the solution on before clearing it for production. =)

Re:Distributed SQLite with Elixir by qaq on Thursday February 16, 2023 @02:48AM (#34816118)
cockroachDB uses Postgres wire protocol and SQL
Re:Distributed SQLite with Elixir by Joel_Mckay on Thursday February 16, 2023 @03:25AM (#34816335)
Cross compatibility is a huge plus from my perspective.

However, there are some designs one should avoid... like auto increment indices etc. =)

Re:Distributed SQLite with Elixir by ammmir on Thursday February 16, 2023 @12:08AM (#34815295)
Manually replicating writes to all nodes seems pretty fragile. At some point you're going to run into classic distributed systems issues, like netsplits, and reconciling the latest count. Maybe a CRDT implementation would avoid these issues alltogether?

But it sounds like LiteFS (with a single writer?) might be a pretty nice setup if it handles replication for you, as you probably don't want to build a Raft or Paxos system yourself.

Re:Distributed SQLite with Elixir by im_down_w_otp on Thursday February 16, 2023 @12:32AM (#34815441)
Litestream isn't for those kinds of use cases. It's for DR, not availability and/or scale.
Re:Distributed SQLite with Elixir by geysersam on Thursday February 16, 2023 @12:44AM (#34815502)
Replicating writes to all nodes does not sound scalable. Is it?
Re:Distributed SQLite with Elixir by simonw on Thursday February 16, 2023 @01:33AM (#34815754)
Litestream is different from LiteFS: same author, different projects.

LiteFS provides replication with the ability to serve read traffic from those replicas. Litestream provides disaster recovery by streaming a constant backup to an S3 bucket (or similar).

Re:Distributed SQLite with Elixir by avinassh on Thursday February 16, 2023 @04:27AM (#34816689)
Whats DR?
Re:Distributed SQLite with Elixir by flowcont on Thursday February 16, 2023 @05:41AM (#34817104)
Disaster Recovery
Re:Distributed SQLite with Elixir by pstuart on Thursday February 16, 2023 @12:39AM (#34815478)
I think the cleaner thing would have been doing event sourcing utilizing OTP and rendered to local SQLite instances.
Re:Distributed SQLite with Elixir by TimWillebrands on Thursday February 16, 2023 @02:21AM (#34815980)
Doesn't event-sourcing require a central datastore where all events are persisted before building the sqlite readmodels?