JS (and py) devs aren't keen on static types... might be a hard sell.
Java seems a perfect fit: static typed, adopted. Also cloud-like "application servers". But already has CORBA and IDL's and serialization - just not popular (why not? maybe goes back to our previous comments) What advantages do you have over that? Uncool, but might be an easy sell.
BTW: There's something tantalizingly puzzling about the whole "JS static types for RPC" thing... not dynamic, not REST... Big picture is not yet clear to me.
REST is popular (eg http://swagger.io to generate APIs and docs); static types mismatch JS, but help RPC (very interesting and popular thread from 3 days ago on "statically linking" microservices - enterprise people concerned https://www.reddit.com/r/programming/comments/3k8sb2/how_we_... )
Yes! I wish there was a way to really define and lock-down exactly what communication/serialisation primitives are required to aid communication between systems. Though I imagine you are right in that we've come quite far with common practices.
Haha - yes the Barrister docs can be a bit confusing - we are getting around to writing a simpler version of them with common examples and use-cases. We really like the schema itself as it does map onto JSON semantics quite nicely.
1) Yep, the messaging format could def be simpler! We just felt that as we're starting out it would be better to stick with standards, even if, as you say, they are nascent. This reduces the amount of things we have to do but also we hope that people much smarter than have thought about the issues involved! Hopefully the client-side bindings will hide most but if needed it's nice to now you can drop down to the JSON format.
2) Hmm, JSON by example, this is super interesting!! I fully understand where you are coming from and it seems so much simpler than JSON-Schema (I was never a fan) Using values as types is quite elegant, and reminds me of some of the type-level programming stuff in the functional circles.
The use of multiple entries in a JSON object seems like a nice way to express sum types - something we are really keen to add to the Barrister IDL (you can extend from an object but I believe there is only a single tree). As you note, although this valid JSON, it may require a custom parser to extract. I'd love to hear more about this technique tho, has it been used elsewhere or is there any further documentation?
3) Primitives - yes, it'd certainly be possible to specify other primitives by encoding them in the JSON string value. Could be risky, as you suggest, when you start looking at regexes and so on - I've not been the biggest fan of defining these as types in the past. Also, how would one go about defining a user-defined type at the equivalent level of a primitive? Super interesting tho, if you have any more thoughts on this I'd love to read them.
I missed the Barrister IDL-JSON binding... kinda important! I must check how they bind choice/sum/polymorphism to JSON...
2) JSON by example schema: Thanks! It's never been used; the above comment is the only documentation. Maybe I should write a RFC... or at least first draft of a spec. And a reference implementation.
Actually using this schema to validate JSON instances requires using all the fields to determine which branch you have. They can have fields with the same name, provided the entire set is distinct. So these are OK (letters as fieldnames):
{a,x}+{a,y}; {a,b}+{a}; {a}+{} (empty object)
But I think JS coders do use polymorphic behaviour (same method names with different code).
3) Add more primitve types. Thinking further, although people end up wanting more precise primitive types for storing data, if JSON is mainly used for transferring values between languages, it really need only be as expressive as the languages themselves, which generally don't go into details like syntactic nature of primitive values and ranges of integers and lists etc. (They wrap these concepts in objects; JSON can too).
I'm stepping back from the idea - I just liked that you could add richer types without losing the property of looking like JSON - but you do lose the property of representing types with the types of JSON values. Can leave the extra sophistication to xml schema (and json schema) for those who need it. Unless JSON itself add more primitive types (which I doubt!).
PS: one more comment to go. I plan to reread your previous comments on CORBA etc now I have a better idea where you're coming from.
Barrister semantically seems like the java type system with minor syntactic differences like field/type order. It uses extends for choice/sum/polymorphism.
In addition to schema definition, it also has interfaces for RPC definition.
The only major limitation I noticed is it lacks recursion (cyclic type references). Important for completeness, but I don't think it's needed that often in practice - I'm interested if this is true.
One great thing about an IDL approach to schema definition, for RPC, is its similarity to objects/classes - because that is the domain it is used from. It's familiar to users, and minimal cognitive burdern to switch between. (Similar argument for JSON - it's an "Object Notation"). In contrast, xml schema looks utterly different from objects/classes.
I think there's even a hypothetical argument for making the IDL look exactly like java - not because it's great but because everyone knows it, even the haters (or, maybe borrow Python's type hint system - or whatever is familiar to your target users).
XML schema has a rich type system, both for structure (eg minOccurs) and primitives. But these aren't needed to represent objects, because programming type systems are poorer and lack those features.
---
I think your main problem is a very old one, of transferring primitive values between different languages (like different languages - even different C implementations - having 16 bit vs 32 bit ints). Text representations like XML and JSON mostly solve this, but of course they can still be too big. You might look at old solutions, like ASN.1, to see the problems that arise, but I think you're better off solving much problems when (and if) they occur - because maybe they just won't come up in practice today because of conventions unconsciously established, even though theoretically they should come up. Just make it as easy to use as possible, to get the thing done that the user wants to do, and you'll get all the feedback/guidance you need.
Maybe you would like to have a type system subset that works for all languages - but this would just hamper those in the one language, or between two languages that are close. OTOH some enterprises like future proof options, in case they need to support other languages.
Would love to hear your thoughts and experience on all these issues.
Hmm, yep recursion not supported atm, although we are thinking of extending the basic system shortly - there are a lot of other features we'd like to add - true sum types, a dynamic operator, and more.
Python annotations are nice, and a lot of users have asked for a way to annotate RPC functions in their code. Doing this all in a cross-language way will be hard and so we're thinking of keeping the external definition in the meantime.
--
Thanks so much for your advice. We're certainly looking at past attempts, and the research in transferring data is vast - I think it boils down to a trade-off between expression and usability - and for now JSON seems to have found pretty good sweet spot. Perhaps with just a simple schema layer on top that will be suitable for most use-cases. I imagine Thrift, ProtoBufs, etc will always beat it on speed, XML (+XML Schema) on expressibility, etc.
For us, as you have guessed, we are just trying to make it as easy as possible for the majority of users – particularly web and mobile developers. This does also mean we're trying to stay language agnostic and find a type subset that works across most languages - JSON primitives let us get some of the way there.
Thank you so much for all your advice - it's been so helpful and we're really happy you took the time out to try it all and record your thoughts. Would love to chat further - please feel free to drop me an email at HNusername @stackhut.com.
What's a "dynamic operator" in this context?
---
> The first RPC was great and CORBA came and went, following this the perception was that it was utterly unsuitable for anything, until perhaps the introduction of ProtoBufs, Thift, JSON-RPC and so on. I personally think it can be incredibly useful, but deciding on just the right features to keep things manageable is incredibly difficult (more so than the tech itself I believe).
[from your grandparent comment] Yes, there's something in this space, but I'm not sure exactly what it is. Historically (there's pre-OOP papers on this), making the network invisible seemed really cool (still does!), but then all the Network Fallacies (https://wikipedia.org/wiki/Fallacies_of_distributed_computin...) got you. I think we've made real progress in general, e.g. JS programmers use async callbacks routinely, and there's promises; http is about dealing with the network. Not only is the tech worked out, but, perhaps more importantly, many coders are familiar with it. Whereas CORBA's superficial design defects (confusing API) obscured its fundamental design defects (distributed objects).
> trade-off between expression and usability - and for now JSON seems to have found pretty good sweet spot. Perhaps with just a simple schema layer on top that will be suitable for most use-cases.
Good summary!
BTW: I personally would like to see your idea usable on a phone - without a full local machine (iterating might be a pain, but your system sounds really fast). Not just serverless, but machineless! A currently underserved niche.
> deciding on just the right features to keep things manageable is incredibly difficult (more so than the tech itself I believe).
I agree implementation is the easier part, though we've been stuck so long, I think there must be a simpler way to look at the whole thing, involving some mathematical or algorithmic insight (as relational algebra did for databases).
Barrister: I sometimes get lost in special cases, and forget the main point that provides the fundamental help to people: Barrister seems feature-full, but from reading the first paragraph or so, it seems to only output docs - because that's all they say it does! If the reader already knows what Thift etc do (i.e. something of an expert, across the field), they could guess that maybe Barrister does more... but the users who just want the main thing you do are easier recruits. Perhaps this is partly why it's overlooked...
StackHut: I'm not sure you need a separate IDL, if it is generated automatically - why confuse the user with it? It seems like a more sophisticated customization tool, that you could leave aside for later? The IDL itself is pretty clear, and although I'd thought about (eg) java classes as defining a schema, I hadn't made the connection that IDL (as from CORBA) also do that.
(1). RPC format. I'm familar with OO serialization and schema languages (unfinshed PhD, book chapters, a library and business), but less so for RPC - so maybe I'm off-base here. And standards - even nascent standards - may be worth complying with. But why not omit the meta stuff, and make it even simpler:
I'm really just wondering if there's a strong reason for the meta data. It can be helpful to orient the reader, but here it is clear from context - the keys and how it's being used. Maybe there are other optional fields you sometimes need?But... for your use-case, perhaps it doesn't matter that much, as the bindings hide it from users (but the point of text protocols is human readable, eg for debugging; so the simpler the better). You could use XML, or a binary protocol.
(2). JSON by example: This is my great idea for JSON schema, which I'm amazed no one has done yet: instead of another meta-format, do it by example:
Because JSON primitive values are typed, you can use a value to signify type. An object therefore also implicitly defines its type (like a java class). For example, the above JSON can also be used as a schema, because it indicates the two nested objects required, and the types of the primitives (string, number, number). Though I suggest a convention of using "", 0 and false as values.
Those zero values help convey that it's a type, not a value. It's temping to want to encode information in the value itself (as opposed to the type), such as a default value - but that maybe a mistake, because there is so much more than can be done with strings than with numbers. Keep it super simple.
[ Arrays are usually not fixed-length, enabling the next trick to encode optional values and polymorphism. ] The JSON spec allows duplicate keys: so you include the same key with different types for all the polymorphic types. Specifically, you include the null value to indicate it is optional. e.g. version is an optional string:
This is a bit dodgey, because all JSON parsers simply return one value if duplicate keys are found. You need to write your own parser. But it is valid JSON - and more importantly, it looks like valid JSON. That's the key idea of "by example" - it looks like what it represents; there's minimal cognitive leap from type to instance.It's rare to want polymorphic primitive types (e.g. string and number), so this is more for polymorphic objects. Unlike the common trick of a "type" field for nominal polymorphism, this is structural polymorphism - where only the different fields distinguishes types. NB: there are some tricky cases here, when the fields overlap, and I'm not sure that client code would want to mess with it.
Finally those non-fixed length arrays: polymorphism is represented by the types of a set of values in the array. In other words, the values aren't ordered, but just represent the permissible types. eg:
That's a schema for an arbitary-length list that can contain instances of those three types of objects, with those mandatory fields, of those primitive types.NB: for a schema of the RPC "header" above, the fixed length array schema represents a fixed length array - a special case.
(3). primitive datatypes: This idea can be extended, in a second level, with explicit primitive datatypes - this is the next level of schema power that everyone wants. Every value is now a string, and looks something like: "url", "date", "email", and then gets closer to XS, with ranges like "int:1..31" and even those ridiculous regex defining valid values (great idea, awful in practice, like the regex for email). The key thing is it still is JSON and looks like JSON, since the datatype specification language is just a JSON primitive value (string), and the syntax and meaning is obvious and familar.
BTW minor typo on your website: s/intergrated/integrated/ (like integer)