News for hackers. Stuff that matters.
Re:How I Wrote a Modern C++ Library in Rust by bluGill on Monday December 03, 2018 @09:23AM (#18588919)
I wish Rust, C++, Go, D... could get together and agree on a common ABI/module format. It doesn't need to be complete (which is to say I'm fine with having to use , just enough that 90% of my program can be written/rewritten in whatever language makes sense and used in the other without having to drop to C.

For starters it needs to have classes (probably using PIMPL like things internally by default). It needs to have some sort of error handling. It needs to support some basic data like std::vector (but they can start from scratch).

Edit: my fingers typed API not ABI first...

Re:How I Wrote a Modern C++ Library in Rust by pritambaral on Monday December 03, 2018 @09:55AM (#18589208)
What you're asking for is a common ABI. To have shared classes, basic data structures, or even error handling, a binary format needs to be defined and adhered to. Of course, to bake all that into languages (and their runtimes), is a great responsibility; but more than that, it can easily be a severe limitation in the way any language may like to do things.

So we have things a little more basic than that, at various levels:

1. API: C. This takes care of a bit more than just data interop and common functions; it maintains an interface, takes care of platform portability concerns (calling conventions, linkage, runtimes), and allows future growth.

2. Modules: From executables and shared objects to services running on a remote machines accessible over TCP/TLS/HTTP(S).

3. Conventions: From free-form (de)serialization formats like JSON/YAML, to read-optimised formats that come with their own implementation code like Cap'n'Proto and Flatbuffers, to share-optimised formats that come with their own read/write/live-share implementation code like Apache Arrow. These come the closest to being a common "ABI", but (thankfully) restrict themselves from being ingrained into a language.

Re:How I Wrote a Modern C++ Library in Rust by jcranmer on Monday December 03, 2018 @02:19PM (#18591961)
> What you're asking for is a common ABI. To have shared classes, basic data structures, or even error handling, a binary format needs to be defined and adhered to. Of course, to bake all that into languages (and their runtimes), is a great responsibility; but more than that, it can easily be a severe limitation in the way any language may like to do things.

Being able to speak a common ABI doesn't necessarily guarantee that your local ABI has to conform identically to it. In Rust, for example, you need to add a repr(C) to be able to actually guarantee that the layout conforms to the C ABI. CORBA IDL is something that tried to achieve this kind of capability.

Re:How I Wrote a Modern C++ Library in Rust by pritambaral on Monday December 03, 2018 @08:05PM (#18594875)
> Being able to speak a common ABI doesn't necessarily guarantee that your local ABI has to conform identically to it.

No, but you still need to support the common ABI. E.g., for Rust, that's an easy job, compared to Go.

Re:How I Wrote a Modern C++ Library in Rust by gpm on Monday December 03, 2018 @09:58AM (#18589229)
> For starters it needs to have classes

But, Rust doesn't have classes.

> It needs to have some sort of error handling

But, Rust doesn't use exceptions, just algebraic data types (`Result`).

> It needs to support some basic data like std::vector

But, these aren't primitives, these are just std library constructs. Which is arguably an advantage, if you get a good FFI you can just use the other language's libraries directly.

A higher level FFI is a good idea, I think you're going to discover that the intersection of languages is surprisingly close to C though.

Re:How I Wrote a Modern C++ Library in Rust by IshKebab on Monday December 03, 2018 @11:36AM (#18590091)
Rust has things that at a binary level are the same as classes, so they could easily be part of a cross-language ABI.

And so what if std::vector isn't primative. It's an extremely common construct that should be part of a cross-language ABI. Hell, it's simpler than strings and they are part of the C ABI.

This is one of those things that everyone is going to nay-say until someone actually does it. The same happened with webassembly. I remember when it was deemed impossible and PNaCl would never work etc. etc.

Re:How I Wrote a Modern C++ Library in Rust by msbarnett on Monday December 03, 2018 @12:28PM (#18590665)
> Rust has things that at a binary level are the same as classes, so they could easily be part of a cross-language ABI.

They are not "at a binary level the same as classes" (whose classes? C++'s classes?), at least not the moment things like vtables get involved.

Re:How I Wrote a Modern C++ Library in Rust by int_19h on Monday December 03, 2018 @06:05PM (#18594071)
Presumably this would be classes in whatever sense the ABI would define them. Which could be defined in such a way that they can be mapped straightforwardly to Rust structs and traits.
Re:How I Wrote a Modern C++ Library in Rust by msbarnett on Thursday December 06, 2018 @06:51PM (#18623730)
> Presumably this would be classes in whatever sense the ABI would define them. Which could be defined in such a way that they can be mapped straightforwardly to Rust structs and traits.

Which would then not map straightforwardly to C++ classes, because the vtable pointer would be in the wrong spot.

You can't have an ABI that makes two things that represent things in concretely different ways somehow resolve that inherent contradiction.

Re:How I Wrote a Modern C++ Library in Rust by hsivonen on Monday December 03, 2018 @09:59AM (#18589237)
> For starters it needs to have classes

Rust doesn't have classes. And as discussed in the article, Rust puts the vtable pointer on the reference while C++ puts it on the pointee, so even fully abstract C++ classes ("interfaces") and Rust traits (when used with type erasure rather than with generics) are implementation-wise different even if they look conceptually similar.

Go doesn't set up the stack in the same way as the others.

Re:How I Wrote a Modern C++ Library in Rust by simias on Monday December 03, 2018 @11:01AM (#18589720)
That could actually somewhat work for C++ and Rust (although getting things like generics/templates across would be a huge challenge and you'd lose Rust's safety guarantees at the unsafe C++ interface. Also good luck dealing with exceptions). For Go and D you have a bigger problem: garbage collection. Having two different runtimes play well with each other is far from trivial and could hurt performance.

The main reason the C ABI is de-facto the lingua franca for language bindings is because it's almost runtime-less. You basically only need to agree on things like stack layout and where the parameters/return values go. It's a super low bar.

Dealing with C++ objects, overloaded functions, namespacing, exceptions etc... Now that's a whole different can of worms. How would you automagically map something like std::cout and its operator<< in Rust or Go for instance?

Re:How I Wrote a Modern C++ Library in Rust by shaklee3 on Tuesday December 04, 2018 @11:16AM (#18600738)
This has somewhat been solved by cgo by having it documented clearly, and turning a runtime flag on that tells you whether you're doing something you're not supposed to.
Re:How I Wrote a Modern C++ Library in Rust by masklinn on Monday December 03, 2018 @01:32PM (#18591472)
> Dealing with C++ objects, overloaded functions, namespacing, exceptions etc... Now that's a whole different can of worms. How would you automagically map something like std::cout and its operator<< in Rust or Go for instance?

You'd define std::cout and operator<< in terms of a sane underlying reader/writer ABI?

Re:How I Wrote a Modern C++ Library in Rust by simias on Tuesday December 04, 2018 @06:02AM (#18598279)
Right, but my point is that then you're generating a complex FFI (which requires high level info about how "operator<<" on std::cout implements streaming and not bitshifting as usual), not a simple binding. Doing that automatically and "standardly" would be quite complex. Those aren't primitive types, they're abstraction built on top of the language's typesystem, g++ is not aware of what a "stream" is in C++, nor is rustc aware of what the Read and Write traits mean.
Re:How I Wrote a Modern C++ Library in Rust by dbt00 on Monday December 03, 2018 @12:12PM (#18590441)
This was the dream of C#, Managed C++, F# and VB.Net on the .Net runtime. It worked out OK for some people, but IMO not all that great -- all the languages trended towards being syntactic sugar on top of C#. The hard part is rarely the language, but learning the semantics of the platform the language runs on, and those languages have very different opinions of what that platform should be.
Re:How I Wrote a Modern C++ Library in Rust by pjmlp on Monday December 03, 2018 @01:35PM (#18591503)
That carries on as UWP, and as someone else pointed to me, Microsoft is going multiple platform with another iteration of it, xplat.
Re:How I Wrote a Modern C++ Library in Rust by lmm on Monday December 03, 2018 @10:12AM (#18589337)
The ABI isn't the hard part. You need to figure out how to track memory ownership when those languages have radically different memory ownership models and often limited support for "foreign"/"unowned" memory.

Passing basic datastructures back and forth isn't hard and there are tools that will autogenerate the code to do that. But when you start using them you'll find they're not solving the hard part of the problem.

Re:How I Wrote a Modern C++ Library in Rust by kllrnohj on Monday December 03, 2018 @01:00PM (#18591093)
No, the ABI is actually the hard part. C++ doesn't even have a stable ABI within just its own ecosystem, ignoring interop with other languages.

Memory ownership is trivial by comparison, particularly in this case when all the languages have the equivalent of malloc/free, so you just make that part of the ABI surface.

Re:How I Wrote a Modern C++ Library in Rust by pjmlp on Monday December 03, 2018 @09:55AM (#18589207)
On Windows it is called COM/UWP.
Re:How I Wrote a Modern C++ Library in Rust by Matthias247 on Monday December 03, 2018 @01:01PM (#18591104)
I think Microsoft might also be working on a newer system with that goal: https://github.com/Microsoft/xlang

I haven't really looked at that in detail, but I would find any tooling for better language interoperability very worthwhile.

Re:How I Wrote a Modern C++ Library in Rust by pjmlp on Monday December 03, 2018 @01:36PM (#18591514)
Many thanks for the heads up, I wasn't aware of it.

Looks interesting, maybe another step into Microsoft Linux.

Re:How I Wrote a Modern C++ Library in Rust by pritambaral on Monday December 03, 2018 @09:57AM (#18589219)
That may be portable across languages and runtimes, but it comes at the cost of portability across platforms.
Re:How I Wrote a Modern C++ Library in Rust by int_19h on Monday December 03, 2018 @06:07PM (#18594088)
There's nothing about COM or WinRT that is inherently unportable across platforms. It's just a standard ABI, that can be expressed in C at that - so any platform that has a C ABI can have either of those implemented. It's just that never really bothered.

GObject is a good example of a truly cross-platform ABI like that.

Re:How I Wrote a Modern C++ Library in Rust by floatboth on Monday December 03, 2018 @10:17AM (#18589371)
GObject (-introspection).
Re:How I Wrote a Modern C++ Library in Rust by nickpsecurity on Monday December 03, 2018 @12:53PM (#18590999)
Interesting enough, the Vala language compiles to C with GObject's for portability:

https://en.wikipedia.org/wiki/Vala_(programming_language)

It was a clever decision. I'm not sure if the right one but a nice thing to try to see how well it works. I wonder if anyone did a "Looking Back"-style write-up on how it's worked out so far with pro's and con's.

Re:How I Wrote a Modern C++ Library in Rust by hsivonen on Monday December 03, 2018 @01:25PM (#18591386)
The GTK+ and GStreamer bindings for Rust are already mostly generated code (and seem to work very well): https://github.com/gtk-rs/gir/blob/master/README.md

I'm not sure what the exact status of extendin GObject hierarchies from Rust is. See https://github.com/gtk-rs/gobject-subclass

Re:How I Wrote a Modern C++ Library in Rust by pjmlp on Monday December 03, 2018 @01:38PM (#18591539)
It works relatively well.

I just have two complaints.

The build times, as Gtk-rs seems to do code generation during the build, and the usual Rc<RefCell<item>> for accessing internal widget data on the callbacks.

Re:How I Wrote a Modern C++ Library in Rust by floatboth on Tuesday December 04, 2018 @04:13PM (#18603646)
Vala is.. good and bad. (Like most things.)

Certainly successful at bringing new developers into the Gtk world (especially thanks to elementary OS and its developer documentation). Familiar syntax for C# devs.

The downside is, well, memory bugs. For example, sometimes perfectly reasonable Vala code results in error messages like "Unable to parse accelerator '\u0008\x8dn\u000b\u0008': ignored request to install 501 accelerators" or downright segfaults: https://gitlab.gnome.org/GNOME/vala/issues/626

Re:How I Wrote a Modern C++ Library in Rust by Vogtinator on Monday December 03, 2018 @10:52AM (#18589629)
QObject (-introspection).
Re:How I Wrote a Modern C++ Library in Rust by pjmlp on Monday December 03, 2018 @10:17AM (#18589363)
You cannot have it all.

Either you get a bytecode format, or a platform specific one.

Re:How I Wrote a Modern C++ Library in Rust by flukus on Monday December 03, 2018 @06:21PM (#18594202)
> I wish Rust, C++, Go, D... could get together and agree on a common ABI/module format.

Don't forget C, with the amount of C code out there you'll have to make sure it can handle this new format. And by the time you limit yourself to the lowest common denominator between all those languages you'd probably just reinvent the C ABI anyway.

Re:How I Wrote a Modern C++ Library in Rust by bluGill on Tuesday December 04, 2018 @08:54AM (#18599398)
The point is the C lowest common denominator is a bit low. I'd like something that by dropping C we can be a little higher.

I'm fine with the ABI having an explicate this pointer (which python already does for classes) which allows C to call/implement classes as well, but it needs to be in the ABI (languages with first class classes can choose to hide the this pointer and v-table).

Operator overloading (which forces name mangling or something like it) is tricky though. As is exceptions. I don't have answers to that.

Re:How I Wrote a Modern C++ Library in Rust by flukus on Tuesday December 04, 2018 @09:14PM (#18605226)
> The point is the C lowest common denominator is a bit low. I'd like something that by dropping C we can be a little higher.

I don't think it really raise the bar much. Rust for example isn't an OO language so classes and v-tables won't be included. Go doesn't have templates so off they go. Operator overloading, multi inheritance, exceptions, garbage collection all go.

What your left with is essentially the C ABI. Being this lowest common denominator is why it's the the common ABI just as much as the success of the language.

Re:How I Wrote a Modern C++ Library in Rust by tolle on Monday December 03, 2018 @09:42AM (#18589084)
Re:How I Wrote a Modern C++ Library in Rust by mhh__ on Tuesday December 04, 2018 @05:20PM (#18604109)
Never ever ever gonna happen, although some progress can be made in areas: D is working (Quite a lot done already) it's way to being ABI compatible with C++.

What could be done, however, is to provide an ABI-as-a-library at each end then pass that with the C interface. Textual Specification, statically processed at each end into some kind of parser type thing. Still borderline suicidal effort, however, as not all languages (Go, I think?) treat the address space equally among a clusterfuck of other things. Any form of parametric polymorphism would be a disaster too, I assume.

Re:How I Wrote a Modern C++ Library in Rust by IshKebab on Monday December 03, 2018 @11:45AM (#18590175)
This is a great idea (ignore the naysayers). It's been discussed before here and in the link 2 comments later: https://github.com/rust-lang/rfcs/issues/600#issuecomment-40...
Re:How I Wrote a Modern C++ Library in Rust by ChrisSD on Monday December 03, 2018 @09:50AM (#18589155)
Reminds me of COM.
Re:How I Wrote a Modern C++ Library in Rust by Koshkin on Monday December 03, 2018 @10:31AM (#18589475)
Re:How I Wrote a Modern C++ Library in Rust by bluGill on Monday December 03, 2018 @09:55AM (#18589210)
Maybe, but COM's reputation is it is too complex. From what I can tell the reputation is well deserved.
Re:How I Wrote a Modern C++ Library in Rust by Const-me on Monday December 03, 2018 @02:21PM (#18591989)
It's too many things at once. The ABI is excellent and is used all the time, see DirectX, MediaFoundation, .NET, UWP, etc, they all use good parts of COM.

Scripting support (IDispatch, WSH) is OK despite a bit outdated.

Registration infrastructure (HKCR), GUI widgets (ActiveX), threading model (these apartments) and inheritance of implementation (aggregation) ain't good, way overengineered.

Compound document format (structured storage), RPC both local and networked, are just horrible.

Re:How I Wrote a Modern C++ Library in Rust by pjc50 on Monday December 03, 2018 @10:26AM (#18589441)
Much of COM's horror comes from being an OO interop system designed to work in non-OO C. All sorts of manual reference counting. If you use COM objects in C# it's much less horrifying, although there's a certain amount of "body horror" involved in firing up one whole windows application inside another one. You can run IE in a cell of an Excel spreadsheet, but that doesn't make it a good idea.
Re:How I Wrote a Modern C++ Library in Rust by reitanqild on Monday December 03, 2018 @11:43AM (#18590157)
> You can run IE in a cell of an Excel spreadsheet, but that doesn't make it a good idea.

Embedding a MS Works worksheet in a MS Works document however was sometimes a good idea in '95.

I've yet to see a better implementation than that and I find it really disappointing that the state of the art has moved backwards in this area (in addition to UX and performance relative to hardware specs).

Re:How I Wrote a Modern C++ Library in Rust by maxxxxx on Monday December 03, 2018 @01:08PM (#18591181)
I agree. Apple also had an effort called "OpenDoc" which they gave up on.
Re:How I Wrote a Modern C++ Library in Rust by maxxxxx on Monday December 03, 2018 @10:49AM (#18589611)
Working with VB made COM really nice. Both for writing COM servers and consuming them. C# is actually a step back because it doesn't do the reference counting right and instead relies on GC to free objects.
Re:How I Wrote a Modern C++ Library in Rust by int_19h on Monday December 03, 2018 @06:11PM (#18594115)
Core COM is basically predefined vtable layout and object reference comparison rules combined with IUnknown for lifetime and type queries. It can be verbose, but it's very simple.

Stuff like OLE Automation is very much not simple, but it's not a requisite part of it all.

Re:How I Wrote a Modern C++ Library in Rust by pjmlp on Monday December 03, 2018 @01:40PM (#18591560)
COM works pretty well in Delphi, .NET, C++ Builder, MFC, C++/CX and now C++/WinRT.

The problems and boilerplate usually affect those that insist in using the C level APIs.

Re:How I Wrote a Modern C++ Library in Rust by mamcx on Monday December 03, 2018 @11:45AM (#18590173)
I have think about this, how a REST API elide a lot of the problems in how integrate disparate stacks...

How have something alike? Maybe put a ring buffer as mediator and push/pull a binary encoding format (messagepack?). Probably need a little of C for the bridge.