Re:The Basics of Apache Kafka
by moedersmooiste on Tuesday September 07, 2021 @10:48AM
(#28445543)
We use Kafka a lot within the company I work for. I think it's great.
The only thing I miss is fast lookups based on some key and/or the ability for subscribers to only receive messages for certain keys.
Re:The Basics of Apache Kafka
by rad_gruchalski on Tuesday September 07, 2021 @01:00PM
(#28447098)
That’s most likely the trade off between the queue and a log. The property of the log is having to look at every message.
The prefixes can be somewhat done with a custom partitioner, the lookup requires another technology. The only way to know the final value under the key is to look at the complete log.
Re:The Basics of Apache Kafka
by rad_gruchalski on Tuesday September 07, 2021 @10:00AM
(#28444961)
This should be labelled as Show HN. One comment - towards the end you say
> You might think that Kafka would be using queue data structure internally. It’s not true. Kafka uses a “log” data structure. It is a persistent data structure which allows only appends, no editing, no deletion. In detail, we will cover this some other day.
Almost immediately followed by:
> Ok, I have a confession . I lied. A Kafka topic is not just a single queue. It’s a combination of queues which helps kafka scale. Every queue is called partition.
Re:The Basics of Apache Kafka
by rad_gruchalski on Tuesday September 07, 2021 @12:03PM
(#28446361)
You’re correct. I have learned something new:
> Off topic: blog posts, sign-up pages, newsletters, lists, and other reading material. Those can't be tried out, so can't be Show HNs. Make a regular submission instead.
https://engineering.linkedin.com/distributed-systems/log-wha...
Reading it was an aha. At its core, the log is the simplest thing that might work in a lot of cases.
The prefixes can be somewhat done with a custom partitioner, the lookup requires another technology. The only way to know the final value under the key is to look at the complete log.
you might look at ksql for filtering purposes or just tried to work such things into your topics/partitions?
> You might think that Kafka would be using queue data structure internally. It’s not true. Kafka uses a “log” data structure. It is a persistent data structure which allows only appends, no editing, no deletion. In detail, we will cover this some other day.
Almost immediately followed by:
> Ok, I have a confession . I lied. A Kafka topic is not just a single queue. It’s a combination of queues which helps kafka scale. Every queue is called partition.
No, it's a simple blog post.
> Show HN is for sharing your personal work and has special rules.
The first line of the rules [1] -
> Show HN is for something you've made that other people can play with. HN users can try it out, give you feedback, and ask questions in the thread.
[0]: https://news.ycombinator.com/newsfaq.html
[1]: https://news.ycombinator.com/showhn.html
> Off topic: blog posts, sign-up pages, newsletters, lists, and other reading material. Those can't be tried out, so can't be Show HNs. Make a regular submission instead.