News for hackers. Stuff that matters.
Re:5 subtle ways you’re using MySQL as a queue, and why it’l... by j_baker on Friday September 16, 2011 @02:14AM (#3003371)
If your queues are getting that backed up, you're either facing bigger problems than your queueing system (most likely workers being down), or you're big enough to afford more machines and/or a custom solution (such as kestrel).
Re:5 subtle ways you’re using MySQL as a queue, and why it’l... by rbranson on Friday September 16, 2011 @03:43PM (#3005986)
The parent post referred to using Redis as a multi-purpose store. If there is a bunch of other non-queue data in Redis and the setup is only expecting it to use 1GB or so, there's likely not a giant amount of room for queue entries left. While everyone is sleeping, some crashed workers combined with broken or poorly configured monitoring can fill up queues very quickly. Been there, done that. Either the OOM hits and there is some data loss or the swap hits and brings everything down.