Lua has been used a lot for that kind of embedding in the past. I think because it's easy to move data between Lua and C? Python would certainly be a big step up but I don't know how hard that kind of integration is.

Lua is a great language for game scripting. Perhaps that’s a better option.

Why is it such a great language? As a developer entirely divorced from the gaming industry aside from occasionally dabbling in modding, lua has always seemed completely inscrutable to me when I have encountered it.

The C API is pretty straightforward and has a stack object to avoid lots of reference counting you would need with Python's c API.

It's also a tiny codebase which is easy to vendor and ship.

Its obscenely easy/consistent memory layout. It makes embedding it into another application straight-forward and easy to manage. Extending the host interface to the embedded scripts is very simple because of this. It's also serialize-able to a degree.

Python is also a decent glue language but creating a Python interpreter is a bunch more work and the API is a bit harder to work with.

LuaJIT is insanely good

It’s a tiny but capable language that’s easy to embed, and can be surprisingly fast with LuaJIT.

I think the main attractions are that it has pretty minimal syntax + core libraries, does a good job binding to C FFIs, and the interpreter is fast.

They have a clean API for embedding. All Lua interpreter state is kept in a struct that you alloc. If you alloc two distinct structs, you have two completely independent Lua vms and you can run them on independent threads if you want to no problem, and so on. (Few language runtimes are clean in this way.)

yes, python is basically impossible to embed in a multithreaded app.

Inscrutable?

The language is _tiny_ (and kept that way on purpose)

Aside from stuff like metatables(which may require you to play around for a bit to understand their value), you can pick it up in a couple of hours. I'm not even kidding. So much so that you see people modifying code without even looking at documentation.

https://www.lua.org/manual/5.4/

https://learnxinyminutes.com/docs/lua/

Lua is most hated lang

It seems reasonable in my ~300 line neovim config :D

and yes; i know that's a ridiculous example that's not really comparable for anything significant.

do you work in games

> Lua is most hated lang

Impossible. Most developers outside the games industry don't even know it exists.

... and if they knew about it, they would use it much more.

Sorry, Lua is not simply not a great language.

The first problem is "Which Lua?" Lua, by itself, is almost completely useless. You first need to compile/install a bunch of things in order to make Lua useful. This has the Perl problem that everybody uses a different dialect of Lua.

The second part is that the Lua constructs for programming in the large are very weak. I believe that Adobe had a postmortem about this (Lightroom, I think?)

And then there is the language, itself. For me, the 1-based indexing is death in this day and age. Sorry. Zero-based is the dominant ecosystem and not fitting into that is simply not acceptable.

> You first need to compile/install a bunch of things in order to make Lua useful.

How is this any different from pretty much any language out there? From a quick eyeball of RustPython's Cargo.toml[1], there are about 70 different dependencies which all need to be compiled. I haven't worked too much with Autoconf, but I am pretty sure CPython has quite a few dependencies.

[1]: https://github.com/RustPython/RustPython/blob/main/Cargo.tom...

> The second part is that the Lua constructs for programming in the large are very weak.

This is deliberate because it forces you to use the tools you are given instead of reimplementing features that can already be implemented by using other primitives in Lua.

> I believe that Adobe had a postmortem about this (Lightroom, I think?)

I can't find this article. Has anyone else had any luck?

If experiences I heard from previous projects(1) (successfully) embedding cython are to go by python isn't necessary good choice for this.

(1): e.g. awesome, blender

I'm curious what you've heard about them that suggests this?

As far as I can tell the maintenance overhead to mapping all the available type from/to python as nicely usable python APIs is a major pain and not just because of cythons current implementation but of e.g. conceptually mismatch between "good" python and "good" C++/Rust/whatever implementations. I.e. you could say that you have a paradigm mismatch between the underlying implementation (of whatever you extend) ad the python API which makes the already hard problem of keeping an API stable even harder.

Furthermore tooling to work in such contexts (as a API consumer) often is very limited or does not exists, but the API producer seldomly has the time/money to produce such tooling either.

Lastly there is the remaining problem of Python by itself being really slow, for it's main use-cases this often doesn't matter as e.g. the scientific computations are run by native extensions python delegates the work, too. But this would mean that you have to also support native extensions support or hope that no extension needs to be really fast (not needing to be really fast is surprisingly often the case) or similar solutions. In turn choosing something which has a good JIT, AOT or similar compilation has benefits.

I think the worst thing about Python and Blender was not necessarily the choice of language, but the frequent API changes and not having anything close to an IDE-like environment.

Not only did old examples often just not work, there was no serious code completion to facilitate creating anything from scratch.

Exactly this. One of the reasons Maya remains favored in studio environments is its strong API stability guarantees. Python is excellent for automation when performance is less of a concern.

Civ4 used Python scripting, and it was a horrendous performance bottleneck :/

The Civ4 core game dll was its own bottleneck. It's is terribly inefficient c++. Years ago, I was working on my own mod to optimize it because I got tired of waiting 15 minutes for the AI to complete a turn on a huge map late game. It was not hard to get it down to just a few minutes. I think it could be used as a poster child for a code base written by someone expecting the optimizer to work miracles. Lots of macros hiding expensive call chains used in tight nested loops, repeated calls in loops to get the same item that should have been called once outside of the loop(s), etc.

Python was not the bottleneck, just the scapegoat for badly written code.

Yes, but (AFAIK) Python itself also became a bottleneck in the Civ4 gigamods, like the (still active) Caveman2Cosmos.

How did you even start optimizing Civ4? Did you replace some exported dll functions with your own? How did you access the state in memory?

Firaxis provided the source for the core DLL for modding. Don't recall exactly when, but it may have been with the Beyond the Sword expansion.

Python is the most incredible language because everything's dynamic. But it comes at an enormous cost - you can't make many assumptions or optimizations easily. It has tremendous utility for many applications. It's this utility and the wide body of existing software packages which makes it so tempting to use as a scripting engine. But it would be a bad move for any scripting that's in some critical region.

It would be nice to have an official, more performant statically typed/less dynamic subset of Python.

Yeah we have mypy, but its not official and not integrated. And subsets that do have a performance benefit (like TorchScript) are niche and/or not sustainably supported.

Well -- other languages have shown up to try and fill this niche.

Not statically typed, but rpython is a statically analyzable subset of python that is used to write pypy: https://rpython.readthedocs.io/en/latest/rpython.html

That looks very reasonable. Converting a Python codebase would be much easier than porting to a new language.

Now all your imports, on the other hand...

I think the greatest idea for Rust scripting is just supporting WASM scripts. I've had some success toying with wasmer in bevy.

You may like Rhai: https://crates.io/crates/rhai