VStack

We have a whole free playlist introducing ECs from a mathematical to practical applications to ECC: https://www.youtube.com/watch?v=sH8ybvIlLvM&list=PL6N_Y7ao_a...

We're working on a certified course that will should be available at https://quantumformalism.academy/courses. It will cover post-quantum cryptography as well.

Interesting point!

Unlock a 50% discount using the promo code: HadamardGate

Hi - we think you misunderstood because you probably haven`t red the post properly. WE will implement this data streaming engine, and give it away as open source ! The reason we are asking people if they can solve this challenge, is for us to know if there are other easy to use data streaming engine that can help solve this as per our relaxed/open specifications.

Hi thanks for the comment! Please check out our free ebook that describes the grid https://leanpub.com/the-grid-vision.

Also please feel free to check out our GitHub repo and take a look at our project Grid Ops (Java) https://github.com/nanosai.

Actually we have been told by another reader recently that Protobuf can in fact encode raw bytes efficiently. We will update the documentation.

Look, why do you even want to use Protobuf? You can just use raw bytes... why even bother with Protobuf - when you can invent everything yourself?

It is pretty clear from the text that Protobuf itself has no way of distinguishing the end of one message and the beginning of the next. Yes, YOU can add that yourself if you want. Like you can add hundreds of other things yourself. But Protobuf doesn't have it.

And - if you really, really love Protobuf so much, let me tell you a secret: You can EASILY embed a Protobuf message inside an IAP message (an ION object field) and there you go... ION works as the message delimiter.

Regarding the "marketing vs. science" comment - We would like YOU to make a comparison of binary protocols that is perfect, solely based on online docs and no access to the core developers... not all docs are equally good, and not everything we have read (e.g.on stackoverflow etc.)seem to be correct.

We don't gain that much by "marketing" a free technology. The customers of VStack.co will be communicating with VStack.co via IAP, regardless of whether IAP catches on outside. We just thought we'd share the work with others.

Yes, we are sure you can build a lot of stuff around Protobuf, but with IAP we will build all that stuff for you.

If that is really true, then we will of course update the comparison page. However, we have put it together from what we were able to find in Google Protocol Buffer's own docs + stack overflow + googling. It is entirely possible that we made mistakes.

Sending schema over the wire is not a good solution for anything else than point-to-point communication. An intermediate node would need every single schema transmitted along with every single messsage, or have another way to keep the schemas cached. That becomes complicated.

The Protobuf documentation says very clearly that you cannot see when one message ends and another begins. Then a protobuf message is not fully self describing. This might be easy to add, but it doesn't have it (according to Protobuf's own docs).

We have looked at Cap'n Proto - but late in the process where we had already looked at quite a lot of formats. From what I can see, Cap'n Proto is pretty much just a binary struct. That is pretty close to what we wanted to do with ION, except we wanted it to be compact on the wire too. We have seen that Cap'n Proto has a compaction mechanism, but we have not yet had time to analyze and compare it to ION's.Cap'n Proto with compaction would be very similar to ION - on a conceptual level.

However, we need to make space for some IAP specific fields coming later in the process (like cache references, column stores and more). Stuff that is IAP specific. That is why we chose to roll with our own encoding in the first place.

Mathias, you also don't need a binary protocol. XML would work. JSON too. But binary is faster and more compact. Same with the table construct. You can work around not having it, but now ION has it built in. You don't make the mistake of serializing an array of JSON objects because you are busy. The objects are serialized as an ION table - not a list of ION object fields. If you ever send an array of objects across the wire with ION (IAP Tools), you will be using the table mode automatically. You save bandwidth and parsing time automatically. Who don't want that - even if you don't need it?

Regarding Copy and Reference, the support for them is still not very good (= not automatic). But imagine your service executes an SQL JOIN query, and in that result a lot of objects are repeated (e.g. same zip + city for a lot of objects). The Copy field can be use to include the zip + city fields just once, and after that refer to them later with a Copy field. That is shorter than including them again. These two fields still need some work to have full support, but we are working on it.

Right now ION is the most well-defined part of IAP. The network protocol itself is still not 100¤ finalized. But, now that we are close to being done with ION (we still have extra fields to add as extended types), we can move forward with the IAP core protocols and semantic protocols. If we do not define a standard semantic protocol for remote object synchronization, IAP will be designed so that you can plug in your own semantic protocol to meet that need.

We believe that while ION is similar in encoding to other formats, it is different in aspects that make ION a better all-round data and network message format. See our ION vs.Other Data Formats Page too.

http://tutorials.jenkov.com/iap/ion-vs-other-formats.html

Not every innovation is from scratch, or 100% unique. MessagePack was designed as a binary encoding of JSON. CBOR too. ION was designed to be a binary general purpose data format able to mimic both JSON, XML, CSV and raw bytes (files), plus the most commonly used data types.

Of course there will be similarities when we try to tackle the same problems, but we also believe we had added something new and useful to the mix.

Please Read this page:

https://developers.google.com/protocol-buffers/docs/techniqu...

If you cannot know where one message ends and the next begins without a schema, it is impossible to use Protobuf messages as network messages when messages need to be routed. This should be pretty straightforward logic. Also, they write that Protobuf is not a good strategy for message bigger than 1MB.

From what we have been able to gather from googling about it, it is because Protobuf encodes raw bytes as var-ints which is inefficient. We have not been able to confirm this - Protobuf is weak in this area of the specification.

Not yet. We have been asked to compare ION to Flatbuffers, Cap'n Proto, Thrift, Avro, Transit, BSON and several other encodings. However, writing the benchmarks and going through the features systematically is a lot of work, so we have not yet had the time to go through them all.

No. That is not correct. You cannot route Protobuf messages. You can only do point-to-point communication with Protobuf. This makes Protobuf unsuited for systems that require routing of messages, like a P2P network where not all communication may be point-to-point but have to be routed through intermediate nodes that do not know the schema of the messages sent. Or, systems that communicate via message queues.

Also, since Protobuf is not good at encoding raw bytes (their words, not ours) we find it unsuitable as a general purpose encoding for messages in a network protocol, where we expect a lot of raw bytes to be exchanged (files, video frames, VoIP frames etc.).

Yes, binary formats are not easily readable in a text editor. But, it is actually possible to convert ION to an XML format and back again without loss of information (we have not implemented this yet). This should make it easier to read messages during debug - especially because you don't need to know the schema for the given message to conver it to XML.

Regarding GZipped JSON, it is true that GZiped JSON is small. But, due to the CRIME and BREACH attacks it is not recommended to compress data sent over encrypted connections (TLS).

If you look at our performance benchmarks page you can see a list of serialized length comparisons. As you can see, as soon as you send a few objects in an ION table, the difference is big. More than what you normally can gain with GZip (except perhaps for String).http://tutorials.jenkov.com/iap/ion-performance-benchmarks.h...

Furthermore, GZip only helps with transfer time, and actually slows down parsing time. If you look at our performance benchmarks you will see that ION parsing time is a lot faster than JSON. Additionally, if you really, really want high speed you do not parse ION (or JSON) into Java objects. You process the data directly in its binary form. If you look at our read-and-use benchmark you can see just how big a speed difference that gives. ION is designed for being processed directly. JSON isn't as good for that purpose.

Finally, ION is designed for fast arbitrary hierarchical navigation. JSON is not.