Re:5 subtle ways you’re using MySQL as a queue, and why it’l...
by rbranson on Friday September 16, 2011 @12:16AM
(#3003181)
What happens when the queues back up and Redis runs out of RAM?
[ Parent
| on HN ]
Re:5 subtle ways you’re using MySQL as a queue, and why it’l...
by j_baker on Friday September 16, 2011 @02:14AM
(#3003371)
If your queues are getting that backed up, you're either facing bigger problems than your queueing system (most likely workers being down), or you're big enough to afford more machines and/or a custom solution (such as kestrel).
Re:5 subtle ways you’re using MySQL as a queue, and why it’l...
by rbranson on Friday September 16, 2011 @03:43PM
(#3005986)
The parent post referred to using Redis as a multi-purpose store. If there is a bunch of other non-queue data in Redis and the setup is only expecting it to use 1GB or so, there's likely not a giant amount of room for queue entries left. While everyone is sleeping, some crashed workers combined with broken or poorly configured monitoring can fill up queues very quickly. Been there, done that. Either the OOM hits and there is some data loss or the swap hits and brings everything down.
Re:5 subtle ways you’re using MySQL as a queue, and why it’l...
by brndnhy on Friday September 16, 2011 @01:55AM
(#3003336)
Good example of a real situation you always end up facing at some point. Why I use ZeroMQ when logging or queueing with Redis.
Re:5 subtle ways you’re using MySQL as a queue, and why it’l...
by justincormack on Friday September 16, 2011 @03:13AM
(#3003471)
But 0MQ can lose items too if the queue fills, how does this help? I can see adding a queue with persistence would work...
Re:5 subtle ways you’re using MySQL as a queue, and why it’l...
by brndnhy on Friday September 16, 2011 @03:28AM
(#3003512)
Yeah, that's true. But my environment is such that any one of 100 or so app servers has a significantly lower chance of running out of memory than the Redis server does.
The 0MQ high water mark is set high enough so that it's virtually impossible not to fix a broken DB by the time messages on the client side create an OOM condition being queued in memory.
Re:5 subtle ways you’re using MySQL as a queue, and why it’l...
by moe on Friday September 16, 2011 @07:53AM
(#3004081)
high enough so that it's virtually impossible not to fix ... by the time
I'll add that to my "famous last words" fortune cookies.
Re:5 subtle ways you’re using MySQL as a queue, and why it’l...
by rbranson on Friday September 16, 2011 @03:52PM
(#3006018)
Ultimately, it's all about the odds. That's what HA, replication, and DR are all about. It's so statistically unlikely for certain things to happen, they just fall out of the realm of reason. Most operations folk I've talked to don't even consider their disaster recovery plans to be within the realm of feasibility. The chances of a catastrophic event rendering the owners of the system defunct is many orders of magnitude more likely than an event that breaks the standard data fail-safes that most datacenters have in place.
Re:5 subtle ways you’re using MySQL as a queue, and why it’l...
by gizzlon on Friday September 16, 2011 @03:32AM
(#3003527)
Unless there are so many items per sec that your persistence can't keep up. Wouldn't this kind of create the same situation: You cant accept all the new items and have to throw some away. Only, now everything is slower. A lot slower.
Ok, the first scenario is caused by the workers being too slow, so it's not exactly the same :)
Re:5 subtle ways you’re using MySQL as a queue, and why it’l...
by justincormack on Friday September 16, 2011 @06:30AM
(#3003919)
Yes, persistent queues have issues, which is why 0MQ exists. But using a non persistent queue to deal with overflow just delays the problem, which was why I asked...