News for hackers. Stuff that matters.
Re:A Celery-like Python Task Queue in 55 Lines of Code by jbaiter on Tuesday February 11, 2014 @11:52AM (#7218541)
Are there any non-distributed task queues for Python? I need something like this for a tiny web application that just needs a queue for background tasks that is persisted, so tasks can resume in case the application crashes/restarts. Installing Redis or even ZeroMQ seems kind of excessive to me, given that the application runs on a Raspberry Pi and serves maximum 5 users at a time.
Re:A Celery-like Python Task Queue in 55 Lines of Code by batbomb on Tuesday February 11, 2014 @01:16PM (#7219192)
My suggestion is to just use RabbitMQ. It's written in Erlang, it uses a BerkeleyDB-like backend for message storage. It's non-distributed, "durable", and optionally with persistent and non-persistent messages. It has a web interface to examine messages. Second suggestion is to use JSON for your messaging format, although it's possible for basic tasks to put all the info you need in the headers.
Re:A Celery-like Python Task Queue in 55 Lines of Code by ris on Tuesday February 11, 2014 @04:22PM (#7220558)
I have personally found RabbitMQ one of the most bonkers, overengineered and painful to manage services I've ever dealt with.
Re:A Celery-like Python Task Queue in 55 Lines of Code by mborch on Tuesday February 11, 2014 @03:07PM (#7220038)
I wrote https://pypi.python.org/pypi/pq, a transactional task queue for PostgreSQL, depends only on psycopg2.

It does about 1000 ops per second, depending on your database configuration, network etc.

Re:A Celery-like Python Task Queue in 55 Lines of Code by dangayle on Tuesday February 11, 2014 @06:49PM (#7221477)
That's pretty cool. What are the benefits of using this over celery?
Re:A Celery-like Python Task Queue in 55 Lines of Code by mborch on Wednesday February 12, 2014 @12:54PM (#7225941)
Celery is typically used with RabbitMQ, which is an advanced message broker.

If you don't need the complexity (most don't), need persistence, transactional guarantees and already have PostgreSQL in your stack, then the pq-library is probably a good option.

It's not really for a distributed setup, but meant to operate within a cluster (local area network, same data center). Most people don't have distributed systems ;-).

Re:A Celery-like Python Task Queue in 55 Lines of Code by asksol on Tuesday February 11, 2014 @12:00PM (#7218599)
I wouldn't really classify this example as a "task queue", it resembles more the RPC pattern really (and I would guess that this example does not persist the jobs in any way). Celery does have a very experimental filesystem based transport that could be used for your use case. I don't know if it fits on a Raspberry PI but celery does not have very high memory/space requirements (c.f. other Python libraries).
Re:A Celery-like Python Task Queue in 55 Lines of Code by jknupp on Tuesday February 11, 2014 @12:02PM (#7218625)
ZeroMQ takes care of the queueing. Though I didn't delve into it in depth in this example, you can create quite sophisticated broker-less distributed systems pretty easily with ZeroMQ.
Re:A Celery-like Python Task Queue in 55 Lines of Code by asksol on Tuesday February 11, 2014 @12:55PM (#7219017)
Right, I understand that ZeroMQ is used to send and receive messages, what I mean is that there's no persistency involved so the tasks will not survive a system restart.
Re:A Celery-like Python Task Queue in 55 Lines of Code by jknupp on Tuesday February 11, 2014 @01:14PM (#7219179)
No, I mean ZeroMQ takes care of the underlying queue for free (there is an underlying set of queues, which can be persisted if necessary)
Re:A Celery-like Python Task Queue in 55 Lines of Code by batbomb on Tuesday February 11, 2014 @01:43PM (#7219432)
No, it doesn't, although it might be possible to sort of emulate it by setting ZMQ_HWM to 1 and enabling ZMQ_SWAP, but I wouldn't bet on it.

The best you can hope is to use the Titanic Service Protocol and just throw data into some sort of disk store. I've looked into doing this, but I settled on using RabbitMQ instead for persistence. Unless you're dealing with more than 10k messages a second, it's just as easy as ZeroMQ.

After a few months of experimenting, I've come to the conclusion that some combination of ZeroMQ and RabbitMQ is likely the easiest solution currently for a combination of low-overhead distributed messaging and broker-assisted persistent messaging.

Re:A Celery-like Python Task Queue in 55 Lines of Code by asksol on Tuesday February 11, 2014 @01:24PM (#7219260)
That must be a very recent development for 0mq then, do you have any source for this new feature?
Re:A Celery-like Python Task Queue in 55 Lines of Code by cschmidt on Tuesday February 11, 2014 @01:45PM (#7219461)
Are you thinking of RabbigMQ perhaps?
Re:A Celery-like Python Task Queue in 55 Lines of Code by jessaustin on Tuesday February 11, 2014 @03:03PM (#7220013)
In a similar situation, I've used inotify to watch things just get dumped on the disk. It's a hack, because race conditions, but sometimes that's OK.
Re:A Celery-like Python Task Queue in 55 Lines of Code by mborch on Wednesday February 12, 2014 @01:44PM (#7226336)
You could use mutex locking on the file system to avoid the race conditions.
Re:A Celery-like Python Task Queue in 55 Lines of Code by joevandyk on Tuesday February 11, 2014 @01:09PM (#7219136)
Re:A Celery-like Python Task Queue in 55 Lines of Code by mborch on Tuesday February 11, 2014 @03:10PM (#7220064)
With PostgreSQL, you can use advisory locks and other little tricks to get better performance, e.g.

https://github.com/malthe/pq/blob/master/pq/__init__.py#L224