Kaitai Struct: A new way to develop parsers for binary structures

Re:Kaitai Struct: A new way to develop parsers for binary st... by lsllc on Thursday March 17, 2022 @08:44PM (#30717459)

Hard to beat the Erlang "destructuring" bit syntax, for example [0] parsing an IPv4 datagram from `Dgram`:

  -define(IP_VERSION, 4).
  -define(IP_MIN_HDR_LEN, 5).

  DgramSize = byte_size(Dgram),
  case Dgram of 
      <<?IP_VERSION:4, HLen:4, SrvcType:8, TotLen:16, 
        ID:16, Flgs:3, FragOff:13,
        TTL:8, Proto:8, HdrChkSum:16,
        SrcIP:32,
        DestIP:32, RestDgram/binary>> when HLen>=5, 4*HLen=<DgramSize ->
          OptsLen = 4*(HLen - ?IP_MIN_HDR_LEN),
          <<Opts:OptsLen/binary,Data/binary>> = RestDgram,
      ...
  end.

[0] https://www.erlang.org/doc/programming_examples/bit_syntax.h...

[ Parent | on HN ]

Re:Kaitai Struct: A new way to develop parsers for binary st... by dhx on Thursday March 17, 2022 @09:48PM (#30717880)

From the page you linked, it appears that the syntax can only handle cases where the size of a data structure is known ahead of reading the data? What about a more complex example of JFIF or ZIP where the parser has to read a data structure of unknown length until terminating bytes are discovered. Do you know of any more complex Erlang examples?

[ Parent | on HN ]

Re:Kaitai Struct: A new way to develop parsers for binary st... by lsllc on Thursday March 17, 2022 @10:17PM (#30718077)

The page covers this further down -- you can "bind" and then "use" a size [0]:

  <<Sz:8,Payload:Sz/binary-unit:8,Rest/binary>>

This bit syntax binds `Sz` as an 8-bit length value, `Payload` as `Sz` 8-bit bytes with the remainder in `Rest`.

[0] https://www.erlang.org/doc/programming_examples/bit_syntax.h...

[ Parent | on HN ]

Re:Kaitai Struct: A new way to develop parsers for binary st... by dhx on Thursday March 17, 2022 @11:35PM (#30718498)

Thanks for the example. Is there a way to perform a positive and negative look-ahead instead of just reading the remaining data to end of input?

[ Parent | on HN ]

Re:Kaitai Struct: A new way to develop parsers for binary st... by eproxus on Friday March 18, 2022 @08:04AM (#30721106)

You also have Erlang’s pattern matching at your disposal:

    decode(<<16#01, Size:8, Payload:Size/binary-unit:8, Rest/binary>>) ->
        [{field1, Payload}|decode(Rest)];
    decode(<<16#02, Flags:16, Rest/binary>>) when Flags =:= 1 ->
        [{field2, true}|decode(Rest)];
    decode(<<16#02, Flags:16, Rest/binary>>) ->
        [{field2, false}|decode(Rest)].

Anything after the when keyword is a guard condition allowing for some manner of positive/negative “look ahead” (Contrived example, but maybe you get the point).

[ Parent | on HN ]

Re:Kaitai Struct: A new way to develop parsers for binary st... by lsllc on Friday March 18, 2022 @09:11AM (#30721746)

Yes, this is exactly how you do it, with pattern matching & guards!

[ Parent | on HN ]

Re:Kaitai Struct: A new way to develop parsers for binary st... by rwmj on Friday March 18, 2022 @04:04AM (#30719662)

My OCaml Bitstring was directly influenced by Erlang: https://bitstring.software/examples/

[ Parent | on HN ]