News for hackers. Stuff that matters.
Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by danielvf on Friday February 02, 2018 @12:58PM (#16292561)
I do some embedded hardware work, and have tests for the C code in python vi FFI. It's also amazing to be able to graph out scenarios with inputs and outputs in Jupiter notebook.

But I hate, hate, hate, cffi's parsing of include files. Makes my life miserable every time I make a change there.

I'm really hoping DragonFFI takes off.

Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by eslaught on Friday February 02, 2018 @01:45PM (#16293057)
This is one of my favorite features of Terra [1]. It also uses Clang to parse the header files, so most things "Just Work" (including, even, JITing inline functions on the fly so they can be used).

[1]: http://terralang.org/api.html#using-c-inside-terra

Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by haberman on Friday February 02, 2018 @03:12PM (#16293830)
Out of curiosity, what do you use Terra for? It looks cool but I haven't found an excuse to use it.
Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by eslaught on Friday February 02, 2018 @03:43PM (#16294046)
I used it to build Regent: http://regent-lang.org/ (TL;DR: Implicit parallelism on supercomputers with automatic compilation for GPUs, etc.)
Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by bliker on Friday February 02, 2018 @01:03PM (#16292606)
I've been using ctypes for the exact same purposes and it has been pretty painless. Why have you decided to use FFI?
Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by danielvf on Friday February 02, 2018 @01:20PM (#16292791)
As far as I can see, ctypes requires you to manually specify everything, rather than load it from the .h include file. This is even more work.
Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by chatmasta on Friday February 02, 2018 @01:53PM (#16293139)
Not to mention another source for error when testing the c code in python. Your test might fail because you forgot to update a header definition in the python, not because you introduced an error in the C code.
Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by kazinator on Friday February 02, 2018 @02:43PM (#16293599)
Though compared with libffi, this can't replace libffi with this in most projects. libffi is a tiny library with no dependencies on Python, Clang or LLVM that's easily integrated into a language, and which doesn't rely on the parsing of any files.

You can statically link libffi for platforms that don't usually have it installed as a dynamic lib; it adds next to nothing to the footprint.

Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by ihnorton on Friday February 02, 2018 @03:40PM (#16294027)
I believe other languages could use this without Python, in principle. It is currently distributed as a Python package to provide functionality similar to the existing cffi package. The Python wheels are 18MB and statically linked -- so no external Clang or LLVM dependencies as far as I can tell. That is indeed quite large relative to libffi (or compared to LuaJIT, which is well under 1MB IIRC). But relative to 100s of MB for a full SciPy stack with Anaconda, it's not that much.

This could also be a foundation for more than "just" C FFI: it includes a full C++ compiler and JIT! Adding C++ FFI support would take a lot of work I’m sure, but I doubt it would noticeably increase the footprint because the bulk is already in LLVM+Clang. Another possibility would be to JIT Cython-generated code on-demand.

Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by chatmasta on Saturday February 03, 2018 @05:05AM (#16296990)
Reading the code, that 18mb is likely from statically linking LLVM into python
Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by kazinator on Saturday February 03, 2018 @11:09AM (#16298131)
ihnorton's comment says this clearly: "... 18MB and statically linked -- so no external Clang or LLVM dependencies".
Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by rurban on Friday February 02, 2018 @04:38PM (#16294408)
This is of course the technically best way, but the immense footprint of 18mb just for the FFI, and the constant c ABI changes in llvm was the reason we didn't do it so far also. You either ship the huge llvm libs or risk an outdated system llvm. But for proper LTS distros it's tempting.
Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by aguinet on Saturday February 03, 2018 @10:48AM (#16298033)
(original author of dragonffi here)

FTR, the footprint of 18Mb is the compressed version, the uncompressed version is ~57Mb. I think there are huge improvements possible here, among them:

* compile llvm with -fvisiblity=hidden

* compile the whole thing with thin lto, which could have the effect to remove unused code

* potentially other idea :)

Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by atgreen on Saturday February 03, 2018 @02:42PM (#16299165)
libffi author here...

Did you consider writing a purpose-built JIT compiler? I believe the FFI use-case is narrow enough that a tiny hand-written JIT compiler would be pretty easy... like the old QEMU template based JIT.

Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by ihnorton on Saturday February 03, 2018 @03:25PM (#16299429)
LuaJIT’s DynASM would go a long way there; might even have reusable stubs. Or slightly crazier, if 520K isn’t too big a burden: wrap the LuaJIT cffi ...from C... and embed all of LuaJIT.

(by the way: thank you for libffi!)

Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by aguinet on Saturday February 03, 2018 @04:43PM (#16299849)
I didn't, as I really wanted to see how that would be possible using only clang/llvm. The way I see it is that it seems a shame to rewrite every possible ABI (and the more it goes the more we have (https://xkcd.com/927/)), where compilers like clang and gcc already does all the work.

One point of my llvm talk of tomorrow is to be able to discuss how we could "extract" these parts from clang to be able to do this the lightest way possible! (but that would still require the embedding of a full LLVM backend which is still huge for the only FFI case).

One good point we have embedding a full compiler is that we can really JIT C code from let's say a python interpreter. The usefulness of this is another debate :)

Said otherwise, I thought the experiment was worth the try, and it seems fun to see how far we can go from here :)

Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by openasocket on Friday February 02, 2018 @03:05PM (#16293779)
I've always wondered if there's a way to do this without needing the header files. Like by parsing the debug information. Obviously, this means you need there to be debug information, but that always struck me as a cleaner solution.
Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by emmelaich on Friday February 02, 2018 @08:01PM (#16295485)
Me too. Surely it would be faster and more portable to put function signatures within the binary lib. No more parsing of source needed.

If you only added the signatures and not all the debug stuff the overhead would be pretty small.

Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by aguinet on Saturday February 03, 2018 @11:01AM (#16298090)
This is actually a pretty funny/nice idea, and could be possible with the actual design!

Using debug info also make the whole thing potentially usable with other languages that emits functions with the good ABI and DWARF informations.

Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by chatmasta on Saturday February 03, 2018 @05:06AM (#16296991)
This should be possible as long as the binary is compiled with its symbol table intact.
Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by onetwotree on Friday February 02, 2018 @03:30PM (#16293956)
Anyone know what "triadic arguments" means in this context? Is it a typo of variadic?

  Some C features are still not supported by dffi (but will be in future releases):
  
     * C structures with bitfields
     * functions with triadic arguments
Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by papercrane on Friday February 02, 2018 @03:45PM (#16294059)
I believe it's supposed to be variadic. I thought it original meant 3-arg functions, but I can see in one of the examples it calls a 3-arg function. The TODO file however says "var args" still needs to be implemented.

* Example calling three arg function: https://github.com/aguinet/dragonffi/blob/master/examples/ar...

* TODO File: https://github.com/aguinet/dragonffi/blob/master/TODO

Re:DragonFFI: C Foreign Function Interface and JIT Using Cla... by aguinet on Saturday February 03, 2018 @10:50AM (#16298044)
this was a unfortunate typo I just fixed: https://github.com/aguinet/dragonffi/commit/dc623098d30f3706...

So it is about variadic arguments. The reason is time, that is I still didn't take the time to make it work, but it does not seem to have big issues doing it!