Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by rad_gruchalski on Friday March 27, 2020 @03:23PM
(#22705949)
Maybe Pulsar gets close. Your general question regarding Kafka reminds me of this: https://medium.com/@rad_g/the-case-for-kafka-cold-storage-32...
I’ve written that.
Edit: kafka jira did not get far: https://issues.apache.org/jira/plugins/servlet/mobile#issue/...
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by PhoenixReborn on Friday March 27, 2020 @04:06PM
(#22706309)
Confluent actually has released "tiered storage" which appears to be a similar feature to what Pulsar has. https://www.confluent.io/blog/infinite-kafka-storage-in-conf...
It's still up in the air as to whether this feature will make its way into the open-source Kafka though.
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by jhgg on Friday March 27, 2020 @07:32PM
(#22707853)
You are looking for Druid: http://druid.apache.org/
I'd love to see a queuing infrastructure that consists of two parallel clusters, running on the same machines:
• a set of write nodes, which each "own" partitions on a topic (in the sense of being that partition's sole writer); where a write-node receives writes to its partition until it fills up a chunk, at which point it shoves the chunk into object storage;
• a set of read nodes, which serve reads and/or streaming subscription requests by 1. streaming archival data by retrieving chunks from object storage into a read-node-local MRU-on-disk-pressure cache; and then 2. once the read stream "catches up" to the end of the archival data, it would switch to proxying to the relevant write-node, which would serve the query/stream from its uncommitted chunk.
-----
I feel like this queue architecture would be useful for all sorts of things, at least for my own use-cases. Especially because it'd scale perfectly-well down to running one read-node and one write-node on one machine, while still being able to make durability guarantees about the committed data. (The durability of uncommitted data, I don't much need to worry much about, personally.)
Is there anything like this on the market?