Re:Startups should use a relational database
by robbles on Sunday December 22, 2013 @04:57PM
(#6951954)
Sounds pretty effective to me. Thanks for sharing this.
Possibly stupid question - how do you retrieve all the expired keys? I was under the impression that once keys expired, they were effectively invisible to clients?
Re:Startups should use a relational database
by AznHisoka on Sunday December 22, 2013 @05:58PM
(#6952160)
So I actually set a soft time to expire as a value for the key. My daemon reads all objects < this time, and deletes those keys itself. The hard time to expire is set to a time much further than this to prevent the scenario of the daemon exiting for some reason, and memory overflowing.
Each object has a field that keeps track of what fields were updated, so I can construct an update query from that.
When I read a row from the DB, I first check if it's in the write through cache, if so I retrieve it there.
Of course, this means that all writes go through my cache. There can't be any other process that updates that table that avoids writing to the cache, or else we have database inconsistency.
This improved the load in my DB server by more than ten-fold, as well as the indexing time to my ElasticSearch server.