gcirino42

It hurts to let it go.. but I love the cause ;)

Google included a complete example of how to (de)serialize data with them, as well as generating the accessor functions for JS. See https://google.github.io/flatbuffers/flatbuffers_guide_use_j...

No, I don't try to decode them directly if I can avoid it. We handrolled a C-based byte packer for our honeybadger project only because function access is relatively slow on the interpreter we have on low end TVs, but reading blocks through a Uint8Array is pretty fast. Writeup is here: http://alifetodo.blogspot.com/2016/05/project-honeybadger-pi... . I can push the C packer to github if there's interest, but since you mention NodeJS, you might have better luck with Paulson's benchmark NodeJS example: https://github.com/michaelbpaulson/flatbuffers-benchmarks

Yup, I'm a fan of protobufs and the more recent flatbufs. Definitely like the tooling around being able to seamlessly switch for readability. I'll check out ion.

I thought my point was clear - don't get involved parsing JSON; I agree with the OP, parsing JSON is a minefield. I went further by implying that it is also unnecessary when ease of reading isn't needed, and called out some alternatives. I think it's amusing that you mentioned protocol buffers - were you aware that when I mentioned flat buffers that they were built in relation to performance inefficiencies in the very protocol buffers that you mentioned?

We didn't just "hackathoned the biggest thing since sliced bread", btw, we took a real world example of exchanging a human readable format for a human-with-tools-readable one and saw a significant win. High-bandwidth also isn't as prevalent as you think, and yes, you're generally paying both performance wise and occasionally monetarily for the laziness you mentioned. But then, if you've known this for 40 years and don't know how to measure it, there's not much I can do for you in a comment.

The correct answer to parsing JSON is... don't. We experimented last hackday with building Netflix on TVs without using JSON serialization (Netflix is very heavy on JSON payloads) by packing the bytes by hand to get a sense of how much the "easy to read" abstraction was costing us, and the results were staggering. On low end hardware, performance was visibly better, and data access was lightening fast.

Michael Paulson, a member of the team, just gave a talk about how to use flatbuffers to accomplish the same sort of thing ("JSOFF: A World Without JSON"), linked in this thread: https://news.ycombinator.com/item?id=12799904

The data used there is cached. We were basically building around having a daughter card handle decoding the video and handling the communication with Netflix servers since there just isn't the memory (and probably not the real time horsepower) to do everything on main CPU. The idea was that an rpi could prime the nametables and communicate via MMC3 style interrupts with the main board, but we wanted to make sure we resisted the temptation of other hacks and simply punch the HDMI jack out of the back of the device and run everything through a primed emulator. We set the rules at "real hardware" and trying to keep true to what you could ship in a cart (though possibly a big one) that a user could put in and play.

Sadly, we had to cut the rpi this time due to time constraints. So we set the data up as if that was doing its job and focused on the current rendering issues and understanding the NES side of things. Hardware interfaces aside (I'm no wiz there), the rpi parts and encoding magic should have been fairly trivial. In the end, we had plenty of CPU cycles left during playback which leave me pretty confident on the rest.

That's not to say we wouldn't have run into some sort of problems with keeping the PPU primed, but we can save that for another (hack?) day.

It's definitely not fake.

and I'm Guy, the other guy ;)

Forgive me, it was a Monday morning when we reshot this. I think the final cart was the original PRGROM + MMC3 / 32 CHRROM banks, so it was probably more like 270k total. The video playback I was commenting on entirely fit into the 256K. In my cursory understanding of the MMC3, I think that's the limit that still fits into a normal mapper 4 and will run on device - I know you can up the PRGROM (add PRGRAM), but that would kill our direct cart --> PPU path for the fast playback.

As for the processor, we were definitely pushing the limit with the design we were coding - we had to back off from glitching quite a bit due to busting the number of cycles we had before PPU. I'm sure we could do better with what we know now, or a lot more assembly tuning, but there was the hard limits on being ready by hackday!

Thanks! Alex had been studying getting better compression by repacking the tiles based on a scheme I still haven't quite figured out, but we were getting CPU --> PPU bound in what we could do during vblank. The MMC3 solution let us basically dma our way around the problems and push a much larger screen update far more often. The original version ripped through playback so fast that we had to slow it down to get a decent number of frames into the available 32 CHR ROM banks (it's a mapper 4 NES rom).

The homebrew community was a fantastic resource! Shoutouts to nesdev, Shiru, and the countless articles we googled!