News for hackers. Stuff that matters.
Re:MessagePack: like JSON, but fast and small by magicalhippo on Tuesday March 10, 2020 @03:29PM (#22538710)
I was looking at MessagePack for communicating to and from my STM32F1-based microcontroller project from the PC controller software I'd be writing. At least the official C library was not optimized for memory usage and code size. I also considered BSON, but it also lacked suitable libraries.

So I ended up using JSON. Yes the message sizes are larger in byte size with JSON but using the jsmn[1] parser I could avoid dynamic memory usage and code size was small. The jsmn parser outputs an array of tokens that point to the buffer holding the message (ie start and end of key name etc), so overhead is quite limited.

For JSON output I modified json-maker[2]. It already allowed for static memory usage and rather small code size, but I changed it to support a write-callback so I could send output directly over the data link, so I didn't have to buffer the whole message. This is nice when sending larger arrays of data for example.

Combined it took about 10kB of program (flash) memory, of which float to string support is about 50%. Memory usage is determined by how larger incoming messages I'd need, for now 1kB is plenty.

A nice advantage of using JSON is that it's very easy to debug over UART.

Though having compact messages would be nice for wireless stuff and similar, so does anyone know of a MessagePack C/C++ library that is microcontroller friendly?

[1]: https://github.com/zserge/jsmn

[2]: https://github.com/rafagafe/json-maker

Re:MessagePack: like JSON, but fast and small by ludocode on Tuesday March 10, 2020 @04:24PM (#22539308)
My MessagePack implementation is designed for embedded:

https://github.com/ludocode/mpack

It can be built to a very small code size, especially when you disable libc, allocations, etc. There are some people using it on embedded devices like Arduino. There's someone working on a port to 8-bit microcontrollers that don't have a 64-bit float, so you may want to look into that as well; see the open issue for it on the GitHub link above.

Re:MessagePack: like JSON, but fast and small by liamdiprose on Tuesday March 10, 2020 @05:07PM (#22539704)
Protobufs/nanopb would be my go-to for minimal message size.

If you want small code size, CBOR seems like a good bet:

> The Concise Binary Object Representation (CBOR) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation. [1]

This [2] C-implementation fits in under 1KiB of ARM code.

[1]: https://cbor.io/

[2]: https://github.com/cabo/cn-cbor

Re:MessagePack: like JSON, but fast and small by RantyDave on Tuesday March 10, 2020 @06:09PM (#22540343)
Re:MessagePack: like JSON, but fast and small by jopsen on Tuesday March 10, 2020 @06:56PM (#22540809)
CBOR is also used on WebAuthn, usage in a web spec means to me that someone smart considered it a sane choice -- and more importantly that the format is here to stay.
Re:MessagePack: like JSON, but fast and small by kbumsik on Tuesday March 10, 2020 @08:30PM (#22541625)
It's great CBOR is accepted in wider area, but I am personally curious why WebAuthn choose CBOR instead of JSON. WebAuthn is a web browser feature, and why W3C would introduce a new data exchange format in their specs? Maybe WebAuthn needed a binary data type?
Re:MessagePack: like JSON, but fast and small by jopsen on Wednesday March 11, 2020 @06:08PM (#22551343)
I'm guessing a binary format is nice when interacting with a device..

Anybody know if (and why) U2F uses CBOR?

Re:MessagePack: like JSON, but fast and small by SlowRobotAhead on Tuesday March 10, 2020 @07:32PM (#22541174)
CBOR’s RFC: https://tools.ietf.org/html/rfc7049

And Amazon has picked it up as a first class citizen in some of their IoT Core features. It’s definitely here to stay.

Re:MessagePack: like JSON, but fast and small by kragen on Thursday March 12, 2020 @11:54AM (#22558090)
CBOR was originally part of the MsgPack project, by the way, before its designer forked it and renamed it after himself.
Re:MessagePack: like JSON, but fast and small by magicalhippo on Tuesday March 10, 2020 @07:13PM (#22540991)
Ah yes I looked at CBOR too, but I dismissed it for reasons I can't recall right now. Will have to take another look.
Re:MessagePack: like JSON, but fast and small by SlowRobotAhead on Tuesday March 10, 2020 @07:29PM (#22541144)
That’s strange because CBOR is almost literally msgpack that got an RFC and has extensions. I cant remember what MsgPack does for online streaming and indefinite lengths.

They’re extremely similar.

Re:MessagePack: like JSON, but fast and small by magicalhippo on Tuesday March 10, 2020 @07:52PM (#22541358)
Looked at it again, seems memory management is a bit of an issue, it supports memory allocation callback but not just handing it a buffer to work with (though I guess allocation should be predictable).

Also I don't know how they got "code sizes appreciably under 1 KiB". On my STM32F1 release mode with -Os it adds about 12kB.

But yeah, maybe I should reevaluate CBOR.

Re:MessagePack: like JSON, but fast and small by SlowRobotAhead on Tuesday March 10, 2020 @11:33PM (#22542728)
For reference, I’m using TimyCBOR because it’s include with Amazon FreeRTOS.

You’re on your own for malloc, which for me is great because FreeRTOS Heap4 management is quite good. So I malloc an object I’m decoding into and parse away.

There are two options parsing arrays and strings/bytestrings and I just chose the option where I specify the pointer to use, vs them using normal malloc then free() later.

I really like this setup. I made a deinit(bad_message) that works anywhere it failed (parse, validate, eval, etc), goes through and looks for pointers that I previously would have malloc’ed.

There is another popular library but I forget what it’s called.

Re:MessagePack: like JSON, but fast and small by kbumsik on Tuesday March 10, 2020 @07:11PM (#22540977)
Yes. CBOR is designed for IoT especially in mind.
Re:MessagePack: like JSON, but fast and small by justinmk on Tuesday March 10, 2020 @05:09PM (#22539722)
> MessagePack ... official C library was not optimized for memory usage and code size

But libmpack is: https://github.com/libmpack/libmpack

- libmpack serialization/deserialization API is callback-based, making it simple to serialize/deserialize directly from/to application-specific objects

- libmpack does no allocation at all, and provides some helpers to simplify dynamic allocation by the user, if required.

- C89

Re:MessagePack: like JSON, but fast and small by osteele on Tuesday March 10, 2020 @04:58PM (#22539608)
ArduinoJson https://arduinojson.org supports MessagePack. I haven't looked at its static or runtime memory requirements.
Re:MessagePack: like JSON, but fast and small by TickleSteve on Tuesday March 10, 2020 @04:17PM (#22539224)
Not messagepack, but if protobuf is ok, then nanopb has given me good results on uC projects.
Re:MessagePack: like JSON, but fast and small by SAI_Peregrinus on Tuesday March 10, 2020 @04:24PM (#22539307)
I'll second nanopb as pretty good. Used it on an STM32F4.
Re:MessagePack: like JSON, but fast and small by SlowRobotAhead on Tuesday March 10, 2020 @07:24PM (#22541091)
It’s good. But it’s a one man show and you are very much in this is how it’s done realm.

I threw away NanoPB in favor of TinyCBOR and haven’t looked back.

Re:MessagePack: like JSON, but fast and small by magicalhippo on Tuesday March 10, 2020 @04:24PM (#22539301)
Ah, that looks pretty spiffy, thanks!
Re:MessagePack: like JSON, but fast and small by jrg_lcsf on Tuesday March 10, 2020 @04:44PM (#22539495)
No idea if it suits your needs, but here's my pet project for microcontroller friendly communication protocols: https://github.com/jean-roland/LCSF_C_Stack
Re:MessagePack: like JSON, but fast and small by matthewaveryusa on Tuesday March 10, 2020 @05:01PM (#22539639)
This library is very small and you need to implement your own I/O via well-defined functions. The parser itself does not use any library (including libc):

https://github.com/camgunz/cmp