News for hackers. Stuff that matters.
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu... by derefr on Friday March 27, 2020 @02:53PM (#22705656)
I've wondered for a while now why there isn't something like a cross between Kafka (durable queue broker) and Datomic (a "database" partitioned into storage chunks, where each chunk canonically lives in object storage and is only cached by the instances doing the queries.)

I'd love to see a queuing infrastructure that consists of two parallel clusters, running on the same machines:

• a set of write nodes, which each "own" partitions on a topic (in the sense of being that partition's sole writer); where a write-node receives writes to its partition until it fills up a chunk, at which point it shoves the chunk into object storage;

• a set of read nodes, which serve reads and/or streaming subscription requests by 1. streaming archival data by retrieving chunks from object storage into a read-node-local MRU-on-disk-pressure cache; and then 2. once the read stream "catches up" to the end of the archival data, it would switch to proxying to the relevant write-node, which would serve the query/stream from its uncommitted chunk.

-----

I feel like this queue architecture would be useful for all sorts of things, at least for my own use-cases. Especially because it'd scale perfectly-well down to running one read-node and one write-node on one machine, while still being able to make durability guarantees about the committed data. (The durability of uncommitted data, I don't much need to worry much about, personally.)

Is there anything like this on the market?

Re:Secor: A service persisting Kafka logs to S3, GCS and Azu... by rad_gruchalski on Friday March 27, 2020 @03:23PM (#22705949)
Maybe Pulsar gets close. Your general question regarding Kafka reminds me of this: https://medium.com/@rad_g/the-case-for-kafka-cold-storage-32...

I’ve written that.

Edit: kafka jira did not get far: https://issues.apache.org/jira/plugins/servlet/mobile#issue/...

Re:Secor: A service persisting Kafka logs to S3, GCS and Azu... by PhoenixReborn on Friday March 27, 2020 @04:06PM (#22706309)
Confluent actually has released "tiered storage" which appears to be a similar feature to what Pulsar has. https://www.confluent.io/blog/infinite-kafka-storage-in-conf...

It's still up in the air as to whether this feature will make its way into the open-source Kafka though.

Re:Secor: A service persisting Kafka logs to S3, GCS and Azu... by jhgg on Friday March 27, 2020 @07:32PM (#22707853)
You are looking for Druid: http://druid.apache.org/
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu... by pram on Friday March 27, 2020 @10:30PM (#22708698)
If the read node ultimately has a dependency on the write node, whats the point of separating them in your architecture?