Re:NewSQL databases fail to guarantee consistency and I blam...
by cdcfa78156ae5 on Monday September 24, 2018 @04:04PM
(#18060937)
> There are a lot of python distributed systems built around in memory stores, like pycos or dask, or built around existing distributed systems like Akka, Casandra, even Redis.
Cassandra and Redis just mean that you have a database-backed application. How do you schedule Python jobs? Either you build your own scheduler that pulls things out of the database, or you use an existing scheduler. I once worked on a Python system that scheduled tasks using Celery, used Redis for some synchronization flags, and Cassandra for the shared store (also for the main database). Building a custom scheduler for that system would have been a waste of time.
Re:NewSQL databases fail to guarantee consistency and I blam...
by cbsmith on Tuesday September 25, 2018 @08:56PM
(#18071821)
> Cassandra and Redis just mean that you have a database-backed application.
Oh there's a lot more to it than that. CRDT's... for example.
Re:NewSQL databases fail to guarantee consistency and I blam...
by pmart123 on Saturday September 22, 2018 @06:25PM
(#18048334)
Well, Celery uses Rabbit-MQ and typically Redis underneath. Rabbit-MQ to pass messages, and Redis to store results.
You can scale up web servers to handle more requests, which then uses Celery to offload jobs to different clusters.
Oh dear... Yeah, that's a terrible distributed system. Interestingly, all the distributed systems I've worked on with Python haven't had Celery as any kind of core component. It's just poorly suited for the job, as it is more of a task queue. A task queue is really not a good spine for a distributed system.
There are a lot of python distributed systems built around in memory stores, like pycos or dask, or built around existing distributed systems like Akka, Casandra, even Redis.