News for hackers. Stuff that matters.
Re:Don't Use an RDBMS for Messaging by DenisM on Saturday September 27, 2014 @04:52PM (#8378174)
One could replace locking with optimistic concurrency. Off the top of my head (MS SQL):

  -- SETUP

  drop table dbo.myqueue

  create table dbo.myqueue (
    itemId int identity primary key clustered,
    worker varchar(32) null,
    myguid uniqueidentifier null,
    work_timeout datetime null,
    picture_to_resize nvarchar(128) not null, 
  )

  -- ENQUEUE WORK

  insert dbo.myqueue(picture_to_resize) values('Kitten.jpg')
  insert dbo.myqueue(picture_to_resize) values('Doggy.jpg')

  -- PROCESS WORK FROM QUEUE

  declare @worker varchar(32)  = 'worker-21'
  declare @guid uniqueidentifier = newid()
  update t set 
    t.worker = @worker, 
    t.work_timeout = DATEADD(minute, 30, getdate()),
    t.myguid = @guid
  from 
    (select top 1 * from dbo.myqueue 
      where work_timeout is null or work_timeout<getdate()
      order by itemId asc
    ) t

  select * from dbo.myqueue where myguid=@guid
  
  -- If nothing found, client should retry...

  -- UPDATE TIMEOUT AS PROGRESS IS MADE
  -- Don't update if another worker stole our job!

  update t set t.work_timeout =DATEADD(minute, 30, getdate())
    from dbo.myqueue t where myguid=@guid and t.worker=@worker


  -- WHEN WORK IS DONE, ITEM SHOULD BE DELETED/ARCHIVED...
Re:Don't Use an RDBMS for Messaging by pmboyd on Saturday September 27, 2014 @06:15PM (#8378363)
I worked at a mass notification company that used a similar approach and it worked well with a handful of workers pulling in batches that could push out 300 to 500 of messages per second and could queue a quarter million messages in a few seconds. There wasn't really a need to optimized past that since the throughput on the rest of the system was the bottlenecked by outside components.

The problem with queuing in {insert new hot queueing system} is that persistent and failover support is often weak. They are by default are more efficient and simpler than RDBMS. If you're running Snapchat, Redis or RabbitMQ or whatever is entirely reasonable.

Re:Don't Use an RDBMS for Messaging by lafar6502 on Sunday September 28, 2014 @05:25PM (#8380941)
Probably unnecessary. You can have concurrency and locking without using any tricks, because SQL Server has this little nice feature - the 'READPAST' hint. It's been added just to make queue implementations efficient and AFAIK only Oracle has something similar.