News for hackers. Stuff that matters.
Re:A Celery-like Python Task Queue in 55 Lines of Code by asksol on Tuesday February 11, 2014 @12:00PM (#7218599)
I wouldn't really classify this example as a "task queue", it resembles more the RPC pattern really (and I would guess that this example does not persist the jobs in any way). Celery does have a very experimental filesystem based transport that could be used for your use case. I don't know if it fits on a Raspberry PI but celery does not have very high memory/space requirements (c.f. other Python libraries).
Re:A Celery-like Python Task Queue in 55 Lines of Code by jknupp on Tuesday February 11, 2014 @12:02PM (#7218625)
ZeroMQ takes care of the queueing. Though I didn't delve into it in depth in this example, you can create quite sophisticated broker-less distributed systems pretty easily with ZeroMQ.
Re:A Celery-like Python Task Queue in 55 Lines of Code by asksol on Tuesday February 11, 2014 @12:55PM (#7219017)
Right, I understand that ZeroMQ is used to send and receive messages, what I mean is that there's no persistency involved so the tasks will not survive a system restart.
Re:A Celery-like Python Task Queue in 55 Lines of Code by jknupp on Tuesday February 11, 2014 @01:14PM (#7219179)
No, I mean ZeroMQ takes care of the underlying queue for free (there is an underlying set of queues, which can be persisted if necessary)
Re:A Celery-like Python Task Queue in 55 Lines of Code by batbomb on Tuesday February 11, 2014 @01:43PM (#7219432)
No, it doesn't, although it might be possible to sort of emulate it by setting ZMQ_HWM to 1 and enabling ZMQ_SWAP, but I wouldn't bet on it.

The best you can hope is to use the Titanic Service Protocol and just throw data into some sort of disk store. I've looked into doing this, but I settled on using RabbitMQ instead for persistence. Unless you're dealing with more than 10k messages a second, it's just as easy as ZeroMQ.

After a few months of experimenting, I've come to the conclusion that some combination of ZeroMQ and RabbitMQ is likely the easiest solution currently for a combination of low-overhead distributed messaging and broker-assisted persistent messaging.

Re:A Celery-like Python Task Queue in 55 Lines of Code by asksol on Tuesday February 11, 2014 @01:24PM (#7219260)
That must be a very recent development for 0mq then, do you have any source for this new feature?
Re:A Celery-like Python Task Queue in 55 Lines of Code by cschmidt on Tuesday February 11, 2014 @01:45PM (#7219461)
Are you thinking of RabbigMQ perhaps?