MQTT vs. Kafka: An IoT Advocate's Perspective

Re:MQTT vs. Kafka: An IoT Advocate's Perspective by ar9av on Wednesday March 15, 2023 @01:58AM (#35164462)

I've been combing the interwebs and following countless tutorials for different types of IoT data gather solutions using various messaging services and brokers for an scale weighment system that I've been developing using VueJS (eventually looking migrate to Nuxt).

So far the concept is simple, a weighment scale has an RS232 (COM) port that streams data out for the tare, net, and gross weights using some kind of micro-controller architecture, Raspberry Pi's, Arduino, PyBoard etc. via some sort of messaging service.

Clients connect to a page where they can "pub/sub" to scales to see the current weight of specific scales and using a remote device. The page acts as a UI to either manage inventory, register consumed ingredients, or register completed tasks.

So far most of the development scripts and test environments I've built all seem to do basically the same thing and will gather the data that's needed but I'm curios if anyone has dealt with anything like this in the wild and if there are any caveats that I'm missing or other technologies that would be better suited for what I'm trying to achieve?

Right now Kafka or RabbitMQ seem to be my main choices for message brokers mostly because they are fairly easy to setup via Docker. If anyone has any recommendation on libraries I should look into that would be awesome! The UI is coming together nicely, I started it in React but switched to Vue3 after I fell in love with the component architecture and composition API.

[ Parent | on HN ]

Re:MQTT vs. Kafka: An IoT Advocate's Perspective by mighty_plant on Wednesday March 15, 2023 @02:43AM (#35164732)

I once had to develop something similar. The solution my company had chosen was to transmit the weights and possible errors via a Websocket connection between the UI and the scale adapter - a small Java service doing the raw socket communication. Weirdly the commands to stop weighing and set tare, get tare, etc. would be send via RabbitMQ from the UI server to the adapter instead of using the WS connection.

[ Parent | on HN ]

Re:MQTT vs. Kafka: An IoT Advocate's Perspective by wmeitzler on Wednesday March 15, 2023 @11:36AM (#35169937)

Having been down this road, I strongly recommend you lock down your requirements before drowning in the sea of transport options. There are a lot of offerings out there that, from the device perspective, basically amount to a reinvention of HTTP(S). Many of these systems exist with the intent to retain the underlying TCP (really, the expensive-to-establish TLS) stream, which is not as much of a concern after HTTP 1.2 retained existing connections.

What are your latency requirements for RS232 to DOM paint? Are you storing time-series data, or just the last value? How fast does the data change, or, when do you want to be notified? Do you have circumstances that periodically disconnect your device from the Internet? It doesn't sound like you have a constrained environment i.e. a cellular connection or battery budget. It also sounds like this is a small-volume project, that you are not building the next Twitter.

If you are interested in historical data, do not overlook the value of generating a larger file on-edge, compressing it, and sending it with a boring-ass HTTP POST. If you want "fast", just set a trigger (i.e. the scale changed by more than 3 ounces in 1 second) to send a small update POST. Periodically perform a GET for config updates, and you're off to the races.

Keeping your messaging layer stone-cold boring enables you to use whatever whiz-bang backend you want, and leverage existing knowledge and tools of the HTTP world. Or just slap it in a database. If you're interested in time series, check out Timescale for Postgres.

[ Parent | on HN ]