Since this is a Lua thread, your reference list should probably start at 1 rather than 0

It is about embedding how easy it is to embed lua, so you should be able to use the host conventions

Oh wow, that does make the non-standard array references rather catastrophic.

you wrap the host across the embedding boundary so that the host and lua can either use their own convention or have access to a special object that explicitely and clearly exposes the other convention.

In general using array indexes as foreign keys should always be considered carefully.

And if all you will be doing is iterating then the starting index doesn't really matter (unless you copy paste loop logic, which is a problem in itself)

Errata: you wrap the host across -> you wrap the array across

Because of this I always thought lists should be iterable/traversable, but have no direct (numeric) index to their items.

Like a stream?

Yes, like a stream, or an iterator. Where you fetch also a reverse iterator if you wish, filter by some predicate etc. A lot of overlap with both relational algebra/sql and with functional programming filter/map/reduce, I guess. Indexes seem convenient, but they're a hack in most cases.

Consider what happens when you sort your array, or delete or insert an item in the middle, or start. Now all your indexes have changed.

React had this issue, and this is why they had to add stable "key" to collections so that when you mutate a sequence of <li/> items for example, it doesn't get confused which is which.

think Set in Java does this

EDIT: Other than the obvious

If you need a few line here an there scripting, sure Lua, JS, PHP... whatever works.

Using these languages for large projects is where the trouble starts: as they push you towards bad solutions that we know are bad for many decades. You need clear description of how to avoid these pittfals, or your growing codebase slowly becomes unmanageable.

I ended up ditching a growing Lua codebase for this reason. Get out before it gets to big to get out.

This opinion gets tossed around a lot, but I've found the opposite to be true, in part because all projects trend towards becoming an unholy mess as they grow large, so being able to accomplish a goal in significantly fewer lines of code has tangible benefits.

If there actually is a correlation between high level languages and "trouble", it may be just because a high level language lets a less experienced developer (or team of developers) get farther than they otherwise would in some lower level language, i.e. the more cumbersome nature of a lower level language forces you to follow best practices earlier or the whole thing never gets off the ground, while the same sloppy devs + a higher level language might actually get as far as shipping something.

Whether that is ultimately better is debatable, but the problem is more how the tool is being used than something inherent in the tool itself.

> all projects trend towards becoming an unholy mess as they grow large, so being able to accomplish a goal in significantly fewer lines of code has tangible benefits

This is a really good point that I didn't realize before - sure, languages like bash and Lua and Tcl might scale very poorly to large codebases, but because they're more expressive than some other languages (coughjavacough), you might be able to implement the functionality you need without needing a large codebase in the first place.

I hear this opinion a lot. What exactly is the characteristic(s) that is missing from a scripting language? IDE integration? The compile-time checking before running? Seems like these problems largely have solutions these days. With stuff like JSdoc annotations or TypeScript, or Teal in Lua land, language servers for most languages etc, unit testing. They might be solved _better_ in a lot of compiled languages, but it's not like it's the Wild West if you're using a scripting language, and you could argue the more modern design in some of them are a fair trade-off for the native compile checking.

With Typescript you need to add an extra compilation step. With that one can just as well not to embed JavaScript but write in the original language.

Teal has an advantage that the typed language can be translated into Lua at runtime so during development it allows for quick prototyping and adding types when the amount of types becomes substantial. But its type system is less powerful than that of TypeScript.

You can use ts-node to avoid the compilation step with typescript.

Or Deno. But only on the back end.

I do not prefer "scripting", but dynamic typing + poor language design.

There are dyn typed langs with langs with nice designs. I like Ruby and IOScript. All lisp-like languages. And there are several dyn typed fit-for-embedded langs implemented in Rust that are very promising. This is very much my own opinion, I know. But reading the article's "cons of Lua" I immediately remembered my fight with that language.

Personally I prefer typed languages nowadays. The stronger the better, as long as there is good IDE support.

I'm not convinced scripting languages are not suitable for large projects. It's said a lot, but I've never seen proof one way or the other. I suspect unmanageable code-bases derive from uncontrollable project forces, ignorance, or not caring, but not because of the perceived short comings of a scripting language.

Indeed. Unmanageable code bases come from conversations like this between management and developers:

“Look, we need you to implement this feature within the next 24 hours.”

“Based on my scrum analysis, I need about a week to implement it.”

“Sorry, if it’s not done within the next 24 hours, you will be out of a job.”

“OK....”

At which point, we get a hideous rush job. I have seen this happen time and time again. There is no language in the world which isn’t going to be able to force clean, manageable code under these kinds of circumstances, which alas can and do happen too often in the caffeine and work obsessed tech culture.

I recently started working at a place which is like this, and the code really shows it. It really does not matter what language is used, it's not going to very maintainable if written so quickly.

Have you coded with Haskell or Elm? Im confident that re-factoring that mess that surely comes out of the 24h sprint will be re-factorable without too much pain in these languages.

Just a feeling, but coding with them feels like they are refactor-optimized langs.

If you keep having to do 24 hour sprints without refactoring it doesn’t matter.

It's a trade off, I think. A compiled static language forces you to behave a certain way at the expense of flexibility/expressiveness, script languages are more flexible, but in a way that is easier to acquire technical debt that might be harder to move away from (IMO).

People do not seems to have any trouble producing unreadable and unmaintainable code in statically-typed compiled languages.

Not at all, but having been on the "dynamically typed spaghetti code with very obscure paths/interactions and no test coverage" refactor wagon, I would take a statically typed option any day.

Indeed the quality to look for is strong typed, not statically typed.

Strong typed code is much easier to refactor.

Unfortunately this requires some 'buy in' from the managers which means that many programs are integer/string typed instead of being strongly typed..

I'm sure with time, the dynlang prototype phase -> static in production will be the norm.

Gradual typing has been improving (might not be the only way but still)

Yes, the problem with strong static typing in many languages is that you have to make decisions about typing at exactly the worst time (at the beginning), while you almost never know up front exactly what you need.

For me the ideal would be a language with the expressiveness and fast dev times of Python, and then type information would be collected from actually running the program during development and would gradually shift from auto-generated type hints to more concrete type declarations that could be enforced but that would also assist in compilation & optimization.

I've read people saying they do that in 70s 80s 90s.. it's probably an unavoidable phenomenon of solution space exploration. Even car makers use gradually more precise models before selling something.

The type gathering from use would be nice.

I find that, with a good IDE, the expressiveness of some typed langs matches that of dyn typed langs. You write a bit more code (the types) when using a typed lang, but you write a bit less tests (the stand-in for types dyn langs offer) and your IDE takes care of many of your type signatures.

The fast dev times, as in quick "edit -> run -> try" loops, it what I find most attractive of the dyn langs nowadays. And gradual typing might be a way to get there, although I think it is not needed (some strong typed langs have decent hot code reload).

I believe mlua [0] is the recommended Lua Rust binding now.

[0] https://github.com/khvzak/mlua

Thanks, I'll take a look

There's also Lua implementation (incomplete) in Rust - Luster[1].

[1] https://github.com/kyren/luster

Some other implementations of Lua:

https://github.com/yuin/gopher-lua Lua in Go

https://github.com/fengari-lua/fengari Lua in Javascript

https://github.com/moonsharp-devs/moonsharp Lua in C#

Why Lua rather than e.g. TCL or Python (both similarly easy to embed, but better-known as full-fledged languages)?

TCL's good for string stuff, but gets very messy if you want to do stuff outside of that. It wasn't really designed originally as a general language: it works and does have some (IMO 'too') clever features, but it has a lot of foot guns as well: comments are actually (almost) ignored procedures, which causes issues, you sometimes have to escape comments or they'll change logic or cause syntax issues (i.e. trying to comment out statements sometimes still triggers syntax errors within the comment!), everything's a string which is great for strings, but not when you need to start validating numbers or similar... TCL makes a lot more sense as a command language (what it was originally designed for - string commands) or a REPL...

Back in the late 80s before Python and Lua were released the following decades, TCL made sense, as it was the only freely available embedable language.

Python's larger and more complete (and I'd argue a better language then Lua), but Lua's compact and very fast as it's a register-based bytecode VM (and luaJIT exists which is even faster) (although if you don't use the 'local' keyword on variables, it's then quite a bit slower as it no longer uses stack-based variables, so the code can end up being more verbose to make it fast), so games commonly used Lua for scripting/gameplay as it was easy to integrate.

Have you seen TCL quadcode? It infers types and uses llvm - Check out the typing diagram; it is kind of crazy. Maybe it is to avoid shimmering more than anything, but I think it is impressive. Thought you might find it interestung even if you don't use TCL.

https://wiki.tcl-lang.org/page/tclquadcode

Let me add my voice to the chorus of posters pointing out that, no, Python is not easy to embed.

Back in 2004 or 2005, Firaxis decided to use Python as their embedded scripting language. They used something called “Boost Python”, a then reasonably easy to embed fork of Python, to embed Python2 in their Civilization 4 gaming engine.

Soon after this, Boost Python got abandoned and Firaxis ended up having to use an outdated version of Python by the time they released their final Civilization 4 expansion.

For Civilization 5, Firaxis instead used Lua, since they wanted an actively maintained code base.

For my own “embed a scripting language in a DNS server” project, I went with a slightly modified Lua 5.1. The entire DNS server, including the Lua scripting engine, is a 103,936 byte sized Windows service. The stack was a little hard to grok at first, but I was able to fairly quickly get used to it and have a Lua script set up configuration for the server, as well as parse DNS queries. [1]

To Python’s credit, the Python2 code used in Civilization 4 is 100% compatible with the final 2019 release of a Python2 interpreter, to the point that I can run map scripts for Civilization 4 -- compiled for x86/32-bit -- on a 64-bit ARM Raspberry Pi and have them generate the exact same maps. Useful when I wanted a particular kind of map for a Civ4 mod, and had to iterate through 300 different random seeds on my Raspberry Pi to find the desired kind of map. After about a month, I had over 180 map seeds meeting my criteria.

[1] https://github.com/samboy/MaraDNS/tree/master/deadwood-githu... for the record

Some corrections: Boost.Python (note the period) was a C++ library for automaticing the generation of cross-language bindings between Python and C++. It wasn't itself a distribution of Python. Boost.Python still exists. However, its successor (pybind11) used features shipped in C++11 to simplify the implementation (and compiles far faster) and is the leading Python-to-C++ binding to use today. I'm using pybind11 in a project today with bog standard Python3 and it works great.

Lua made inroads into the game dev community in particular thanks to LuaJIT. Despite it actually being a fork of Lua that hasn't kept up with changes in the base language, LuaJIT remains popular for its speed.

Python is all but "easy to embed" - for one thing it's huge, and it requires linking to native libraries which may conflict with your own (openssl comes to mind). It also has an unstable ABI (and even API in some cases). Lua, in contrast, is just a single, very lightweight DLL, with no external dependencies and a stable ABI.

Yeah, Python is "theoretically" easy to embed, that's why it uses the GIL

In practice, it is hard to embed and we get the GIL disadvantages, so that is not working out so fine.

The GIL is less about python being easy to embed, than it is about embedding other libraries into python. The GIL was set up to make it easy to wrap C code and expose it to python code. A task for which it served quite well at the time and to be honest lasted longer than I would have expected.

My impression is that Python is not actually easy to embed (and I say this as a fan and frequent user of Python).

Modern tcl is big. Embedding something like Jim is still possible, but I think Lua is more popular than tcl and has a more approachable syntax if you want laymen to use your DSL.

Modern Tcl may be big, but it is not too big to embed. I know, because I embedded it into a Go system at a previous employer. Unfortunately, it is proprietary, so I may not point you to the repo, but it required remarkably little glue code.

I also evaluated Python and Lua. As others have noted, Python appears to be a right royal pain to embed. I actually had more experience with Lua prior to that project, but that experience leads me to believe that Lua is, generally, the wrong language.

Looking back on the project, I would not hesitate to do the same thing again, ideally as open source so that other Go projects can embed a scripting language. It got the job done, and did it well. Non-programmers on our team were able to write both configuration and logic successfully & productively, and apparently enjoyed the experience. Tcl itself ended up being a pleasure to use. While I personally would have enjoyed something like Embeddable Common Lisp, I think that would have been to much to ask of the rest of the team.

Interestingly, the four languages Lua, Python, Tcl and Lisp each can be said to take one idea and run with it. Lua is everything-is-a-hash-table (well, almost everything); Python is everything-is-an-object; Tcl is everything-is-a-string; and Lisp is everything-is-a-list (well, in theory: in practice it is really everything-is-an-object). I don’t know if this says anything deep about scripting languages, but it is at least interesting — right?

Python is not easy to embed. It's very unsecure and is difficult, if not nigh-on impossible, to effectively sandbox. (see RestrictedPython)

It's my understanding that Lua can be more easely sandboxed, at least compared to Python.

Lua is smaller, and one of the fastest in the bytecode-land. It also used in a large number of diverse applications [1], so in terms of popularity, I think it has nothing to be ashamed of compared to others.

[1] https://en.wikipedia.org/wiki/List_of_applications_using_Lua

When I say "embed" I mean: add the source tree to your project and compile statically (see the CMake project for Lua 5.4).

To my knowledge, you can't do that with Python. I don't know about TCL though.

Not sure about TCL, but python is much bigger, has a bigger memory footprint, and is a bit slower than LUA. Also, check how easy is to integrate Lua into your codebase

I embedded Python and it was not easy to embed at all.