“Land initial Rust MP4 parser and unit tests”

Re:“Land initial Rust MP4 parser and unit tests” by pjc50 on Thursday June 18, 2015 @03:47PM (#9741173)

A very complex piece of software is exactly the sort of thing you want to avoid writing in C or assembly. The last person I know writing a codec implementation did it by machine-translating the spec into a functional program that output assembler, C, python, Verilog etc as targets. That did require handcrafting "leaf nodes" for things like matrix multiplication, but they were small enough to verify.

[ Parent | on HN ]

Re:“Land initial Rust MP4 parser and unit tests” by haberman on Thursday June 18, 2015 @03:57PM (#9741234)

Yes, I think of Rust as a promising target for parser generators. Thinking about text parsing, if Bison generated Rust, then you'd have a memory-safe parser that should be about as efficient as C. Something like this probably already exists or is being worked on. Ideally existing code generators could target Rust also to get memory safety.

There are some SIMD optimizations in parsers though that I don't know how easily you could express in Rust. The quintessential example of this for text parsing is Clang's optimization that uses SSE to skip over C++ comments 16 bytes at a time:

https://github.com/llvm-mirror/clang/blob/61f6bf2c8a8e94c4fa...

[ Parent | on HN ]

Re:“Land initial Rust MP4 parser and unit tests” by steveklabnik on Thursday June 18, 2015 @04:22PM (#9741390)

I am far from an expert in this area, but you can do SIMD with Rust: https://github.com/huonw/simd

[ Parent | on HN ]

Re:“Land initial Rust MP4 parser and unit tests” by haberman on Thursday June 18, 2015 @04:47PM (#9741523)

This looks like a good start (though experimental). If it supported something like AltaVec's vec_any_eq() intrinsic on its u8x16 type, that would do the trick. vec_any_eq() takes a vector and a value and returns true if any element of the vector equals the value.

On x86 with SSE, this could generate a sequence of two instructions: pcmpeqb (do 16 byte-wise compares) followed by pmovmskb (collect the 16 comparison results into a single byte). Then you'd get the same efficiency as what Clang does (Clang searches 8 bytes at a time for a '/' character when skipping over comments).

[ Parent | on HN ]

Re:“Land initial Rust MP4 parser and unit tests” by jroesch on Thursday June 18, 2015 @07:50PM (#9742285)

Huon is working on SIMD full time at Mozilla this summer as far as I know. We should see more mature support materialize in the next couple of months for sure.

[ Parent | on HN ]

Re:“Land initial Rust MP4 parser and unit tests” by buster on Friday June 19, 2015 @05:45AM (#9743788)

I have been working with https://github.com/kevinmehall/rust-peg for a while. Also there is https://github.com/Geal/nom

[ Parent | on HN ]

Re:“Land initial Rust MP4 parser and unit tests” by nitrogen on Thursday June 18, 2015 @10:50PM (#9742763)

A while back there was a blog post (I think by Dark Shikari) about the use of assembly in x264. I couldn't find it by searching, but basically the very best C versions of essential image processing algorithms were orders of magnitude slower than the hand-crafted assembly versions, especially with SIMD instructions.

To create a viable codec, a functional spec compiler would have to close that performance gap. Edit: it might be possible to create a set of SIMD algorithmic building blocks (e.g. like liboil[0]) that could be verified for correctness then incorporated into the compiler.

[0] https://wiki.freedesktop.org/liboil/

[ Parent | on HN ]

Re:“Land initial Rust MP4 parser and unit tests” by teacup50 on Thursday June 18, 2015 @11:18PM (#9742834)

There has been work on typed assembly that could also apply: http://research.microsoft.com/en-us/projects/talproj/

[ Parent | on HN ]

Re:“Land initial Rust MP4 parser and unit tests” by peteretep on Thursday June 18, 2015 @11:32PM (#9742881)

Conveniently computers are getting orders of magnitude faster and cheaper, while the software on them - by virtue of complexity - has an increasing attack surface.

I will happily trade a video player utilising 4% of my CPU power and with fewer potential vulnerabilities than one using 0.4%.

[ Parent | on HN ]

Re:“Land initial Rust MP4 parser and unit tests” by nitrogen on Friday June 19, 2015 @12:38AM (#9743028)

When it comes to video encoding, the tradeoff is often between encoding live video or not encoding live video.

I think there's also an important tradeoff to be made in how much of a carbon footprint we dedicate to software. I really don't think we should be trying to make software expand to consume all available resources. We have to be both efficient and secure.

[ Parent | on HN ]

Re:“Land initial Rust MP4 parser and unit tests” by Too on Friday June 19, 2015 @01:08AM (#9743101)

Even when you are on a mobile device constrained by battery?

[ Parent | on HN ]

Re:“Land initial Rust MP4 parser and unit tests” by RBO2 on Thursday June 18, 2015 @04:11PM (#9741332)

Very interesting. How can we get more info on this? Is there any public code or product out there?

[ Parent | on HN ]

Re:“Land initial Rust MP4 parser and unit tests” by pjc50 on Thursday June 18, 2015 @05:12PM (#9741643)

The resulting commercial product is not the codec itself, but a set of test signals for codec verification for use by hardware vendors.

http://www.argondesign.com/products/argon-streams-hevc/

[ Parent | on HN ]

Re:“Land initial Rust MP4 parser and unit tests” by RBO2 on Friday June 19, 2015 @12:46AM (#9743053)

Thanks! I'm contributing to an open-source project where we do a lot of standardization (including MP4). We're trying to improve the parser generation by using a model from the specification. We even have some funding for this. I'd be happy if we could discuss this ([email protected]). Better standards means a better world for everyone :)

[ Parent | on HN ]

Re:“Land initial Rust MP4 parser and unit tests” by fithisux on Friday June 19, 2015 @06:45AM (#9743943)

Can you imagine, writing a Bluetooth spec (and profiles) in a high level language and let Rust (or something else) produce the native driver code?

[ Parent | on HN ]

Re:“Land initial Rust MP4 parser and unit tests” by RBO2 on Friday June 19, 2015 @03:53PM (#9747238)

Yes that's exactly what it is about! Unfortunately, our current result shows that it still requires much time to model the spec correctly. Any help appreciated, my contact is available on the message above :)

[ Parent | on HN ]