Your custom engine sounds interesting, though. I can recognize some thoughts ..
---
Decoda uses your PDB to locate the Lua C library within your .exe and the Win32 API CreateRemoteThread to inject the code into your process to hook the Lua API.
I was the tools lead for "Superman Returns: The Videogame" (the PS2/X630 Superman game, no the N64 one). We did the same thing. All of Metropolis was essentially "modeled" in Maya with plug-ins handling importing and exporting between that and the in-game format.
It was an open world streaming game and it would have killed an artist's machine to try to load the entire city into Maya, so the plug-in would let an artist pick which chunks of the city to load, load them in, let them make changes, and save the results back out.
It worked out fairly well.
Scott Bilas also documents doing something similar for Gabriel Knight 3, one of the “grandfathers” of data-driven engines and bit of an influence on ECS.
Instead of excel however it was comma-separated text files that shipped with the game. During development could edit the files locally and reload the data without recompiling.
Since we’re in a web heavy forum: An sql database and a web editor would also allow for a distributed workflow.
Ok maybe a sheet is fine for a state machine if you don't care about visualizing transitions... Why reference cells when you could have named functions in a source file that could actually parse the syntax? That file would be trivially merged. You want to be loose on types so I guess its fine but you could actually have type safety and linting all the other language features and tooling if you just used a file full of named globals.
The mind boggles
In systems where we used this it was common to have 100+ states x 100+ events. Visualizing that would have been fun, but not useful.
One implementation had layerable sheets. So, you could define a base state sheet and multiple overlay sheet that the game could enable dynamically on major changes.
Within each event column there would be long, broken up runs of "do the same thing" for runs of states. So, you could define a chunk of code in a cell below the main table and reference it in the runs and it would show in the main table.
Almost all functions were trivial. 1 or 2 statements. "Modify a value. TransitionToState X". There would be hundreds of these little snippets in a sheet. Naming each one would double the complexity for no gain. Let alone having function signatures. They'd all be `void randomName(event)`. And, Lua's a dynamic language. You aren't going to get much error checking in an IDE. Better to just keep everything local to the sheet rather than flipping between Excel and VSCode.
And, in the implementation discussed above, most of the cells were full of numeric/string values or references to assets.
https://github.com/luau-lang/luau/ https://github.com/sssooonnnggg/luau-debugger
I'm working on an engine based in C++, Luau, and OpenGL - started almost 2 months ago. I aim for it to me MIT license open source, but it's too early for sharing. When it is, I do plan to post a show HN with the Github link.
Then we added "State machines as Lua exported from Excel". Rows are states, columns are events, cells are code to execute given a state+event combo. I've done this a few times. It makes huge state machines manageable.
Our games were very stats-heavy and our designers liked Excel. So, a new thing we added as "Dynamic data sources as grids of Lua exported from Excel". So, fill out Excel sheets like normal. But, instead of Excel script, every cell is evaluable Lua code. Strings, numbers, bools, functions are all values in Lua. So, a cell might contain a number. Or, it might contain a function checking the contents of two other cells and optionally triggering an event on some other object.
We shipped multiple games on a single executable using this system. Artists could lay out 3D scenes and 2D UIs with hooks for the Lua to control it. And, the designers could populate scenes and UIs from Excel dynamically according to the state of the game. The programmers mostly worked on game-agnostic features in C++, and the heavier side of scripting in Lua for game-specific features.
Aside: Lua is being so dynamically typed makes it not great for large-scale software engineering. But, https://teal-language.org/ might be a good TypeScript-For-Lua. I haven't tried it. Also, are there any Lua debuggers newer than the ancient https://github.com/unknownworlds/decoda that require zero integration? Decoda just need a pdb of your executable and it can automatically debug any scripts passing through the Lua library.
Eventually, we switched to Unity for corporate reasons. As a primary implementer of our custom engine, I think the switch was overall a good thing. Our games had started to outgrow what our little engine team could deliver. The artists reported they felt slightly less productive working in Unity's editor. I switched roles to finding and working around the undocumented bugs in Unity that we ran into. Then later finding which bugs had been fixed without being mentioned in the release notes so I could delete my work-arounds. It was a very boring couple of years until the company burned to the ground because of the same corporate reasons that lead us to switch to Unity :P