Re:MQTT vs. Kafka: An IoT Advocate's Perspective
by sally_glance on Wednesday March 15, 2023 @08:53AM
(#35167540)
Got it - I think for the first 1-2 years one broker instance should be plenty. So the decision to up the complexity can probably be postponed to the point where it becomes clear that we hit 10M+/lots of brokers will be required.
Re:MQTT vs. Kafka: An IoT Advocate's Perspective
by rad_gruchalski on Wednesday March 15, 2023 @09:13AM
(#35167834)
Definitely. Focus on solving the business problem first, grow the business. When you have the need to make a switch, you will find a way forward by rebuilding your your MQTT infrastructure to fit your scale. If you ever need to.
Re:MQTT vs. Kafka: An IoT Advocate's Perspective
by sally_glance on Wednesday March 15, 2023 @03:47PM
(#35173748)
Yeah - business is smart home renewable energy stuff, not lightbulbs. If they don't go international real fast the small oldschool broker cluster solution will probably be plenty for the next 5 years.
You have 10 brokers, each broker handles 1M connections. You have a TopicA subscriber on broker 9 and 10, someone publishes a message to TopicA on broker 2. Your broker 2 needs to forward the message to brokers 9 and 10 but no others because if you do this for every message, you waste a ton of compute. So you need a method to distribute your connection table: which brokers currently have connections with subscriptions to TopicA. This is reasonably easy to embed in the broker for a relatively small number of brokers but if you need hundreds of brokers, your global connection table will not fit on each broker. Scale this up - millions of topics and millions of connected clients. You need a way to offload the connection table somewhere else, find a reliable method to distribute changes to your connection table. This is where we have successfully used Kafka - distributing the connection table and state changes.