News for hackers. Stuff that matters.
Re:MQTT vs. Kafka: An IoT Advocate's Perspective by mrkeen on Tuesday March 14, 2023 @04:01PM (#35157513)
I jumped onto https://mqtt.org/ to try to answer my usual use-case question about non-Kafka messaging, which is: "Do the messages get saved anywhere so you can come back and read them later?" Still not entirely sure about it.

But I did see:

    This is why MQTT has 3 defined quality of service levels: 0 - at most once, 1- at least once, 2 - exactly once 
I'm a big fan of advertising the impossible on the front page.
Re:MQTT vs. Kafka: An IoT Advocate's Perspective by avereveard on Tuesday March 14, 2023 @04:15PM (#35157746)
> advertising the impossible

eh if you read the finer print it's just a deduplication id appended to every message. blog doesn't go into detail on what happen when two client pust a message with the same it, or what happens if there is more than one failure (i.e. client fails to detect a service outage and during the service outage the message is consumed by the broker but persisting fails) but in general the usage of a at least once + a deduplication id is not something revolutionary.

Re:MQTT vs. Kafka: An IoT Advocate's Perspective by alecthomas on Tuesday March 14, 2023 @04:17PM (#35157773)
> I'm a big fan of advertising the impossible on the front page.

Do you mean like Confluent do? https://www.confluent.io/blog/exactly-once-semantics-are-pos...

Re:MQTT vs. Kafka: An IoT Advocate's Perspective by codeflo on Tuesday March 14, 2023 @04:23PM (#35157879)
The impossibility of "exactly once" is a theorem, not an opinion.

Knowing that, the article you linked is funny. It begins with maybe a thousand filler words complaining that all of this is "poorly understood", and it's not "impossible", just "very hard". Then it gets to the meat: Yeah well, you know, it's not quite "exactly once delivery", just "exactly once semantics", and to achieve that, messages need to be idempotent so that duplicates don't matter.

We all know that. It's called "at least once".

Re:MQTT vs. Kafka: An IoT Advocate's Perspective by simplotek on Tuesday March 14, 2023 @04:37PM (#35158106)
> The impossibility of "exactly once" is a theorem, not an opinion.

It's quite likely that your definition of what "exactly once" means differs from the one followed by MQTT. As this issue was documented years ago, I doubt this is a relevant argument to have, unless we want to feel smart by criticising others.

https://www.eejournal.com/2015/05/28/is-exactly-once-deliver...

Re:MQTT vs. Kafka: An IoT Advocate's Perspective by ryanjshaw on Tuesday March 14, 2023 @04:50PM (#35158334)
That's the Kafka Streams API. "Exactly-once semantics" has a very specific meaning in the context of that particular API, which the article could probably do a better job of clarifying upfront. (Otherwise it is an excellent overview of the problem and solution provided by the Streams API.)
Re:MQTT vs. Kafka: An IoT Advocate's Perspective by alexisread on Tuesday March 14, 2023 @05:32PM (#35159099)
With mqtt it depends on the broker, eg. Emitter.io can save them for a week etc. Offset for a client is usually stored on the broker, so if a client reconnects, all of the messages it has missed are forwarded to it.

As mentioned in other answers, service levels have a defined meaning, which is different to the absolute theoretical meaning, and really is to do with the message aks.

Really, as the article mentions, kafka and mqtt are for different purposes, with some overlap. Kafka is all about the log, whereas mqtt is about uncertain connections. A better comparison which I've yet to see, is comparing mqtt to nats.

Lastly, kafka is much easier to administer using redpanda, which doesn't have zookeeper, combines the registry and kafka connect (see WASM runners) with the runtime, and has a very nice console for debugging.

Similarly, Emitter.io does a great job with clustering for mqtt.

I'd like to see an open source kafka-mqtt bridge that worked in both directions as they all seem to go mqtt->kafka only.

Re:MQTT vs. Kafka: An IoT Advocate's Perspective by imglorp on Tuesday March 14, 2023 @05:53PM (#35159441)
nats +1
Re:MQTT vs. Kafka: An IoT Advocate's Perspective by jon-wood on Tuesday March 14, 2023 @05:33PM (#35159118)
MQTT isn’t designed as a persistent log, but can fulfil some of what you might want to use one for.

Each message has a couple of flags, the first being Quality of Service, which as you quoted above determines deliver guarantees. 0 is fire and forget, with potential loss of messages. 1 will queue messages for delivery to offline clients that are subscribed to a topic (within reason, all brokers set limits on that), and 2 is often described as “exactly once”, but is in fact just a more involved dance to acknowledge messages.

The other flag is a Retain flag, which instructs the broker to associate that message with the topic it was sent to, and send it on to any newly subscribing clients when they subscribe. This is good for use cases like remote device configuration - you can send it to a topic, setting the retain flag, and then when a device comes online it’ll immediately receive new configuration.

MQTT is great as a message queue for remote devices, mostly because it’s so lightweight anything with an IP stack can integrate with it, but I’m not sure why anyone would attempt to make it a piece of core infrastructure.

Re:MQTT vs. Kafka: An IoT Advocate's Perspective by jonquark on Tuesday March 14, 2023 @05:59PM (#35159546)
MQTT.org can't answer that as it's a web page for for a protocol. I've worked on platforms that do have a historian feature but it will vary from broker to broker.

(disclosure, I work on Eclipse Amlen and it does not - but people often rig it up to a subscriber that funnels (some/all) messages into databases