Right now, my project's rather small 100 kLOC codebase compiles (with very minimal #ifdef hackery) on Android, iOS, Mac, and Linux; but Windows is still very much a WIP, even some of my third-party libraries don't compile on VC. I'm actually considering trying the MINGW toolchain at this point, and I'd be curious to hear anyone's thoughts?

I'm sure part of it is that the compiler itself probably developed with the tight feedback loop from developing and maintaining large codebases in house at Microsoft (like Windows and Office). It's pretty hard when part of the internal pressure to support large codebases like that conflicts with the need to conform to outside third-party standards. I've heard great things about the compiler people at Microsoft, and I'm sure they have a technically strong team, but they are most likely caught in the middle of this organizational deadlock.

I'm sure it doesn't help that I'm not mentioning any specifics, I'm going to be revisiting the Windows desktop port of my mobile app soon (which I gave up and haven't touched for a couple months), and everything will be fresh in my mind again. I do vaguely remember something about having to explicitly add more #include statements to pull in header files that were already getting pulled in by my other compilers.

The big problem was Wine, which can behave quite differently to real Windows in some places. It's certainly not bug-for-bug compatible. So I kept finding myself having to boot into Windows anyway for debugging, so eventually I just installed an msys development system on Windows with gvim and used that.

There was also something else wrong with mingw for Linux, but for the life of me I can't remember what. There was something about mingw being different to mingw32? Missing headers? It's been too long.

I've run into a difference between the two recently. It turns out that MinGW implemented POSIX glob() post-fork, which hasn't been ported to MinGW-w64.

[1] As I side note, I found that MSYS2 makes a huge difference in how you develop software for Windows with a POSIXy toolchain. Anyone who uses MinGW on Windows should try it.

- Do you guys have an RPC library written in anything other than C++? If not, could you point me to protocol specs so I can start writing my own?

- Since it uses a streaming model to support random access, what encryption method do you think would work best with Cap ' n Proto that would keep it speedy and still retain all functionality?

Thanks!

People have written implementations in Rust, Go, and Erlang, and wrappers around the C++ library in Javascript and Python: https://capnproto.org/otherlang.html

Scroll down that page for some info on how to start writing an implementation in another language.

The RPC protocol spec is here:

https://github.com/sandstorm-io/capnproto/blob/master/c++/sr...

> - Since it uses a streaming model to support random access, what encryption method do you think would work best with Cap ' n Proto that would keep it speedy and still retain all functionality?

Hmm, I'm not clear on what you mean by "streaming model" -- I think of "streaming" as the opposite of random access.

Regarding encryption, this is a very big question and there are a lot of different needs and use cases to consider. Mostly I don't think that use of Cap'n Proto affects encryption decisions much, but if you want to make sure you don't lose random access, you should of course use a cipher that supports random access, like chacha20 or AES-CTR.

Right, sorry, I re-read my comment and confused myself too. Seems like it's bad for me to go on HN without a fresh cup of coffee (it's 10am now here in the Philippines).

Thanks for your swift response, I'll experiment with AES-CTR (since I'm more familiar with it than chacha20). And thanks for pointing out that there are wrappers for Python/Go already, the programming language I use daily and was thinking of building libs for! Again, great work, and I'll stay posted.

https://download.libsodium.org/libsodium/content/secret-key_...

(Also, a lot of the issues we face affect headers, which need to be compiled into the client projects.)

I've also built another message format recently (yes I know, they are never ending). It can't do everything Cap'n Proto can do, although it shares a lot of the same values. One thing I chose to do is to have order preservation on types, which can be very useful. It does mean that the wire format is largely BE though. Anyway, that's an aside.

I'm curious what you think of Amazon ION by the way?

Good stuff!

Consider javascript talking to common lisp. Of course JSON has a canonical mapping to javascript, but it does not for common lisp. Should a JS array be a lisp list or vector? Should lisp's NIL be false or null? Should a JS object decode to an alist, plist, or hash-table? &ct.

For many years I was in the schemaless camp before JS came along. Then for a number of years I was in the self-describing camp because I was thinking that if we don't accept JavaScript and JSON are pretty fundamental on the web we're fools and everyone seemed to be passing around JSON. So in that period was thinking that MsgPack was pretty damn good.

Recently I've switched back to the schemaless view, but with strict order preservation and richly typed fields. Very "tuple" based... so works well with Lisp and JavaScript but also C++. Highly inspired by Linda.

I don't do what Cap'n Proto does and lay out the fields and all that good stuff so that you can kind of memory map it onto structs.

That is nice and I understand the motivation for sure, and I have worked on systems that do that in the past with very good results, but currently my thinking is that compactness without additional compression is a good balance.

Also, since the protocol is order preserving (where it matters) you can do radix sort operations or hash maps on the server side extremely quickly. That was the ultimate motivating factor.

[SomeDateTime, "hello", 1.0f, [10,20], "Foo", false, [SomeMatrix]] etc etc...

Inner tuples or BLOB's are length prefixed of course so you can skip them, but basic types and strings are not. Strings are zero terminated while Ints/Floats/Doubles are BE and complement encoded to preserve sorts, and also Integers are packed to minimum size.

Memory mapping is possible on this system too, but it's very "functional"... there's no attempt at pointer preservation. I don't go that far and Cap'n P seems to be preserving some of the semantics of ProtoBuf at least in that regard, which I'm sure is a good thing for many scenarios.

You could still do that with what I'm doing but it would be at the application level. Same with self-description actually, you could easily build something that looks like JSON if you wanted... either:

[["foo",1.0],["boo","cat"]...] etc

or

[[1.0,"cat"],["foo",boo"]

the choice is really up to the application programmer.

That might be a bit "loose" for a lot of people to stomach, but it works very well for what I'm doing, has a lot of flexibility and packs really well

Like I was suggesting, protocols are something of an art and I don't think we're at a final solution yet, which is why many people are constantly inventing new ones :-)

Hopefully we will get to some consensus one day!

Really hope that it can be fixed at some point, but not holding my breath on that one.

I think it will take a major effort to reform that format although I am hopeful now that we at least have UInt8Array and friends that are starting to expose a broader set of machine friendly types.

The thing is that it is least common denominator, and when you are dealing with high perf, cross language systems, it really isn't a good wire format or storage format.

It takes ages to parse, it's lossy, lacks commonly used types (or you have to annotate it with non standard attributes)... or worse guess the intention, and it's pretty verbose.

But again, that said it is a widely used standard and one that we have to live with. So there is that.

I have this crazy idea that there's a middle ground: self-describing and schemas?

We can kind of glue this together (there's lots of json schemas floating around out there now, it seems), but it would be awfully interesting to see these well-supported as a pair (with explicitly language agnostic schema definitions -- which seems to be a sticking point for most of the json schema strapons).

</tangent>

https://github.com/sandstorm-io/capnproto/blob/master/c++/sr...

(There are also similar libraries for Protobuf.)

(I use CBOR a lot -- I'm otherwise quite happy with it!)

EDIT: I guess there's a "CDDL" listed on the tools page, but... It's still single implementation (ruby) and I don't see a clear link to a grammar for it.

I'll definitely give it a whirl. Cheers!

Cross platform libraries that really only care about a single platform is an endless source of frustration. :(

We rejected Cap'n Proto for the same reasons. I don't see a need to use bleeding edge C++ features when the same(or better) results can be achieved with code generation.

Kudos on the effort but if you're looking for wide adoption you've missed the mark.

Well... We're not, actually. We're looking to support the needs of Sandstorm.

It's nice if my code is useful to others, and if people want to contribute better Windows support I'll be totally happy to review and merge those changes! But wide adoption of Cap'n Proto (outside of the Sandstorm context) is not part of our business model, unfortunately.

(Note: The poster you were replying to isn't associated with Cap'n Proto nor Sandstorm.)

There are a few major reasons I didn't go that route:

* The .proto language has a lot of weird quirks that I don't like. Some of the quirks are specific to the protobuf encoding (e.g. int32 vs. sint32 vs. fixed32 being different types), while other quirks have no particular rationale behind them. I didn't want that baggage.

* The .proto language does not treat interfaces (aka services) as a first-class type. That is, you cannot define a field whose type is an RPC interface type -- a reference to a remote object. The ability to do this is a critical part of Cap'n Proto's interface design.

* It's highly unlikely that the protobuf team would be interested in accepting changes to the language which were not actually supported by protobuf. This means that if I shared the language, I would have my hands tied when it comes to new features -- or I'd also have to implement Protobuf equivalents to make them happy.

The reasons you gave make sense, although I'm not super familiar with the RPC aspect of protocol buffers so that's new to me.

(Note that Sandstorm is focused on making Cap'n Proto work well for Sandstorm. We welcome contributions, but we generally don't have resources available ourselves to work on third-party feature requests unrelated to Sandstorm. That is, unless you want to pay us a bunch of money, in which case, feel free to contact me. ;) )

https://github.com/nemoNoboru/capnp-ruby

https://rubygems.org/gems/capn_proto-rpc

The FlatBuffers encoding is based on vtables and is relatively straightforward (the runtime library is tiny). This also means it's inefficient for small messages, but in my testing its vtable deduplication worked great for my use case (~100k messages of the same type per memory-mapped file), in that the vtable overhead tends quickly to zero.

Cap'n Proto has a more complex encoding that is probably more efficient in terms of wire size, and particularly for small/standalone messages, but the runtime is larger as a result.

Cap'n Proto is both. I.e. you have the serialisation, which can be used on its own, but you can also define interfaces with methods that can pass those serialised structs as parameters, and which return asynchronous promises.

Right now I can only see a way to create lists, so access will be in linear time.

I thought of the hashtable, but that means I have to keep the hash-table in RAM. By contrast, the entire capnproto structure itself could be memory mapped, thus not having any RAM constraints.

I'm thinking larger datasets here, large enough where overhead of e.g. JSON becomes a problem with RAM (on mobile devices), but not yet large enough to have to use SQLite.

I might be overthinking it, and could move to SQLite straight away. Just trying to keep my stuff as simple as possible. Conceptually simple, that is. Capnproto is conceptually simple: a tree dumped to disk, allowing on-demand memory mapped access to requested parts.

I'm saying you create a capnp list, and then you store elements into the list at position according to their hash -- i.e. how you'd build a hashtable, but the capnp list itself is the backing array.

You would have to do this at write time, of course, and make sure the hashing is consistent between runs.

I'd still prefer the framework do it for me. It seems quite involved.

Thanks for taking time to answer my questions!

I see the classes: ArrayOutputStream/VectorOutputStream, i assumed these are the ones you are discussing.

However, I am obviously very biased. :)

In general, because I felt there was too much resistance to me pursuing ideas that I wanted to pursue. Google has become increasingly top-down, whereas it was fairly bottom-up when I started in 2005. That's not necessarily a bad thing, but I felt that I personally would be happier running a startup where I could call the shots.

FWIW, if you just want to write code, be comfortable, and make a crapton of money, and you don't care if you're implementing someone else's ideas, I highly recommend working for Google. That's not meant to be sarcastic or disparaging -- I totally respect that approach and there are days when I wish that were me. But if you have ideas of your own and you won't be happy unless you see them implemented... it probably won't happen at Google.

(To be fair, some people at Google would surely argue that my problem is that my ideas are crazy and bad, and I don't have any firm evidence -- yet -- that they are wrong.)

FTFY

It may not be that your ideas are crazy and bad, but rather that your sane good ideas simply aren't appropriate in the context of advertising.

Enough with the Stockholm syndrome. There are pleanty of perfectly great ideas that Google would never pay their employees to work on, however explicable or inexplicable the reason might be.

>A blow for mobile advertising: The next version of Safari will let users block ads on iPhones and iPads [1] : [...] An Apple realist might argue that its great rival Google makes more than 90 percent of its revenue from online advertising — a growing share of that on mobile, and a large share of that on iPhone. Indeed, Google alone makes about half of all global mobile advertising revenue. So anything that cuts back on mobile advertising revenue is primarily hurting its rival. (Google has been less friendly to adblockers than its “open” positioning would suggest.)

[1] http://www.niemanlab.org/2015/06/a-blow-for-mobile-advertisi...

Your post was snide. Snide remarks are commonly downvoted on HN, regardless of their accuracy.

Regarding your follow-up, you're taking a very simplistic view of Google that doesn't really capture reality. Google obviously builds lots of technology that doesn't directly affect advertising revenue. Most decision-makers at Google are not asking "how does this affect advertising revenue" for every decision. You'd know that if you ever worked there.

Several reasons, but most critically:

The phrase "advertising company" generally refers to a firm in the business of creating advertising. Google's main business is an online media company selling advertising space (both in its own advertising-supported services and alongside online media provided by others.)

That sounds like a pretty major advertising company to me.

Can you name a bigger advertising company than Google?

If they're actually a technology company driven by the demands of their users and striving for technical excellence, then do you expect Google to support built-in ad-blocking in Chrome and Android like Apple already supports it in Safari and iOS any time soon?

To what extent are such disagreements purely technical (e.g. byte-oriented wire encodings vs. roughly-like-memory) and to what extent are they "political".

By "political" I am talking about your claims about decentralisation and diversity, as if you are selling Sandstorm as the IBM-PC for the cloud millennium: a neutral platform any old software developer. That is very different from Google's way of doing doing business.