The simple fact of introducing a one-million-row read for the latest data of each "pixel cell" is fairly insane. You must have a cache for such data. "I'd still have have Redis cache, though" is not even debatable. It doesn't have to be Redis, but is definitely has to be a cache of one kind or another.
place=> explain analyze select * from board_bitmap ;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------
Seq Scan on board_bitmap (cost=0.00..14425.00 rows=1000000 width=6) (actual time=0.009..57.295 rows=1000000 loops=1)
Planning time: 0.160 ms
Execution time: 90.510 ms
(3 rows)
And, with triggers from an activity table, the entire write operation can be atomized so there aren't any race conditions.I don't think you understand how fast Postgres is on modern hardware. What took a large cluster 5 years ago can be done on a single system with a fast NVMe drive today. We really might not even need Redis in this situation.
And, yes, I have to deal with viral content, so this is right up my alley.
No, just using it to store the update log.
But I don't know if there's any obvious problem with querying a handful of megabytes once per second either.
I lean towards just using the ratelimiting stuff we already have in place (via memcached, which we talked about in a previous post). We just overlooked it.