News for hackers. Stuff that matters.
Re:Using Lua coroutines to create an RPG dialogue system by SomeCallMeTim on Tuesday January 10, 2017 @01:16PM (#13367288)
I've done this kind of thing in Lua before, but what troubles me about it is the fact that a lot of the state of your game ends up encoded in the Lua stack.

What happens when the user wants to save? There's Pluto [1], which is somewhat slow for complex state. And it doesn't work on LuaJIT code -- and I tend to like to run games on LuaJIT.

And it's opaque: If you're debugging some kind of interaction in the game, you can't just look at the save state to see what's where.

I used to be a big fan of Lua, and this kind of AI and behavior coding was a component of that. But ... I think that, on balance, actually creating a data format that represents the state and "coding" using that plus state machines is actually a more robust and useful way to accomplish the same thing.

[1] http://lua-users.org/wiki/PlutoLibrary

Re:Using Lua coroutines to create an RPG dialogue system by jamesu on Tuesday January 10, 2017 @03:50PM (#13368856)
An additional problem is what happens when you need to patch the game? If you need to modify some form of quest or dialogue script written as a coroutine, how do you migrate the state? What if you've changed the script 5 times in a series of patches, and the user is still on version X?

Then again you still have this problem even if you power your quest/dialog with data instead.

Without sufficient information it's practically impossible to make a smooth migration.

Re:Using Lua coroutines to create an RPG dialogue system by leggomylibro on Tuesday January 10, 2017 @04:59PM (#13369525)
That is definitely an issue with this sort of thing; I think that you'd need to organize your conversations in a tree structure, and have 'save/load' methods for each node that could transcribe between a serialized data format like json or yaml, and functions in lua.

With a format like that, you have the option of writing your dialogues in code or plain text, and easily switching between the two. The only major downside is that you can't easily define arbitrary behavior that way; every unique action that a dialogue event could trigger would need to be transcribed in your 'save/load' methods, which may feel limiting if you have a lot of hard-to-generalize logic.

Re:Using Lua coroutines to create an RPG dialogue system by buzzybee on Tuesday January 10, 2017 @02:59PM (#13368382)
On the other hand, it's certainly more engineering to have to build a custom format. I implemented a story engine that compiles to behavior trees, and then subsequently to a small instruction set in this past year and, well, we're talking about three months to go from having no technology to having a pipeline capable of complex scenarios. But I came out of it with a much more competent strategy for addressing assets than anything I've done in the past, so I'm not complaining. It is the core technology of the game(choice driven, text simulation) so it needed a big investment. And the game is going to have content updates, which makes save games terrifying; In the end, I designed it so that it has episodes and restarting the game starts it from that episode so that I don't have to worry about massive amounts of persistent state getting permanently corrupted.
Re:Using Lua coroutines to create an RPG dialogue system by gravypod on Tuesday January 10, 2017 @03:18PM (#13368554)
You could do this in C with something like libdill as they give you a handle to a coroutine. All you need to do then is create a state_t that holds the coroutine reference managing the object, the channels for it, and all of the state needed for the computation. During saving you prune dead and save only the computation state. During loading you recreate all of your Objects.

It wouldn't be all that slow and I think it would get over most of the overhead in some implementations I've seen of UI/Interaction code for games.

Re:Using Lua coroutines to create an RPG dialogue system by SomeCallMeTim on Tuesday January 10, 2017 @04:48PM (#13369427)
I actually hacked together my own "coroutine" scripting library in C once upon a time. Lots of macros. Same problem though.

When I said slow, I meant slow to save the entire state of the Lua VM. You can't even do that in C, so it doesn't really apply to the problem I was pointing out.

Re:Using Lua coroutines to create an RPG dialogue system by gravypod on Wednesday January 11, 2017 @12:31AM (#13371988)
All saveing your object state should be is memcpy'ing your array of all objects into some sort of static memory and on load looping through every object and starting a coroutine for every object. I don't think that's too much overhead.
Re:Using Lua coroutines to create an RPG dialogue system by SomeCallMeTim on Wednesday January 11, 2017 @03:09PM (#13377087)
Sorry, but you're totally missing the key point.

Say you have an AI that's made 3 decisions and you're in the MIDDLE of a coroutine where it's waiting for the next criteria that will shift it to its next state. So the AI code looks like:

   local player = lookForAPlayer()
   if navigateToPlayerLocation() then
      local quest = givePlayerQuest();
      while (!quest:done()) do
        waitWithStandardReply("Waiting for you to finish the quest!");
      end
      if (quest:resolved()) then
        giveQuestReward();
        while (true) do
           waitWithStandardReply("Thanks for finishing that quest for me!";
        end
      else -- quest abandonded        
        while (true) do
           waitWithStandardReply("Thanks for nothing!");
        end
      end
   end
This is rough example code, and not how I'd probably handle this case specifically, but it gives you the general idea. Each of those functions (except the quest member functions) has, somewhere inside it, one or more "yield()" statements, and some logic to make its goal happen incrementally.

Yes, it's awesome to be able to code AI logic that way. But how do you save the coroutine state? The actual execution state of the coroutine itself, including what line it will resume next, the values of any bound variables, the values of any local variables, are all part of the state of the AI in that case.

In C, you've got functions that may be relocated somewhere else in RAM and potentially stacks to save (if you're using a "real" coroutine library), so just doing a memcpy won't work. What would you memcpy to save the line that the script is paused on? In Lua, it can be done with Pluto, but then you can't use LuaJIT. I see libdill has the ability to use a user-supplied stack, but it doesn't appear to have a "resume" call that takes such a stack memory block, so the best that would happen is that the coroutine would start over from the beginning. Meaning that if you saved the above coroutine in the middle of a quest, the character involved would offer you a new quest when you saw them instead of accepting the result of a successfully allocated quest.

If your object states are all being handled in traditional objects, then you're not using the coroutines in the way described by the article.

Re:Using Lua coroutines to create an RPG dialogue system by stevekemp on Tuesday January 10, 2017 @01:26PM (#13367411)
I've certainly hit situations in the past where the Lua stack became too large, and caused all kinds of problems in debugging.

But despite that I think coroutines have their place, and this was a nice example that I found when I was looking for inspiration.

(I've been working on a Lua-scripted mail client for the past couple of years, and I'm starting to think more seriously about async-behaviour.)

Re:Using Lua coroutines to create an RPG dialogue system by SomeCallMeTim on Tuesday January 10, 2017 @04:53PM (#13369471)
Coroutines are awesome, don't get me wrong.

BUT, contrary to what I believed when I was still working with Lua, making the coroutine explicit (like async/await in C# and JavaScript ES2017) is actually even better.

In particular, when your coroutines are explicit, you can say "queue up this task, this task, and this task, and continue here when they're ALL done." So when you're dealing with several tasks that might take different amounts of time to complete, they can do the work in parallel.

I went from being a major Lua fanboy and posting frequently to the mailing list to drifting away from involvement to abandoning the language, in a relatively short period of time. I even wrote a blog entry about it. [1]

I still think Lua has better "bones" than JavaScript, but with TypeScript, async/await, and JavaScript JIT engines everywhere, TypeScript just wins for me.

[1] https://realmensch.org/2016/05/28/goodbye-lua/

Re:Using Lua coroutines to create an RPG dialogue system by daurnimator on Tuesday January 10, 2017 @09:43PM (#13371329)
The problem with making coroutines (like async/await in JS) is http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...

This is one of the reasons I love lua's coroutines.

Re:Using Lua coroutines to create an RPG dialogue system by unscaled on Wednesday January 11, 2017 @06:04AM (#13373118)
I assume this is exactly what SomeCallMeTim meant. Yes, explicitly awaitable futures/promises are slightly less ergonomic than coroutines, but this is a very reasonable trade-off.

You give up: 1. Extra cognitive load every time you need to await. 2. You've got to be more careful when designing some generic higher-order functions (you need to decide which color their function arguments should be).

In return you get: 1. Explicit concurrency. No function that you call unexpectedly suspends behind your back. In some cases this explicitness could be actually important (e.g. when using per-thread/process shared memory). 2. Futures/promises/tasks are first-class values. You could pass them along, save them for later, await on all or any of them and so forth. 3. Theoretically speaking, promises can be more memory efficient than coroutines since you don't need to allocate a full stack for each of them - just the state you need.

"What Color Is Your Function" is misleading: there is no strictly superior solution for this tradeoff.

Futures/promises encode concurrency as a monad and like every monad they are viral and once you start propagating them around, they can be a pain. But the alternative for not encoding some concern of your program as a monad is to let it pass through an implicit side-channel (I/O, error-handling, global state, built-in scheduler) that is harder to control and can't be treated as first-class object in the language.

Go itself chose to go the explicit with error-handling. It doesn't even uses Monads, so it gets the worst of both worlds in my opinion, but the designers' main concern with exceptions which they managed to avert is still valid: with (unchecked) exceptions you never know which function you call is going to raise an error on you somewhere down the line.

So yes, with Go you'll never have with unexpected throws, but you'll have to deal with unexpected goroutine suspension. In Go all of your functions will be happily colorless with regards to concurrency but will be colored with regards to error-handling.

Again, both choices here are valid, but they are not so brain-dead simple or universal as this article is trying to put them.

Re:Using Lua coroutines to create an RPG dialogue system by SomeCallMeTim on Wednesday January 11, 2017 @03:25PM (#13377215)
Very well put, thanks.

Explicit promises have very nice, composable behaviors that I'm exploiting all the time, and that have no equivalent in the Lua coroutine world.

And once you have propagated them around, I think the ergonomics are just fine; I've taken some pretty hairy callback logic that no one could fully understand without careful study and turned it into async/await code that pretty much anyone could follow.

I think the implicitness of "this could suspend" does frequently cause serious problems; in Lua this shows up as the dreaded "Attempt to yield across a metamethod C call boundary" [1] bug. Talk about cognitive load! If you're calling C from Lua, and that C code then calls back into Lua, and the latter function yields, you get this error. So you have to be very careful about what is allowed to yield -- and since any function in Lua, at any point in the call stack, can yield, it's easy to accidentally shoot yourself in the foot.

[1] http://stackoverflow.com/questions/8459459/lua-coroutine-err...

Re:Using Lua coroutines to create an RPG dialogue system by anonymoushn on Wednesday January 11, 2017 @06:09PM (#13378319)
> 2. Futures/promises/tasks are first-class values. You could pass them along, save them for later, await on all or any of them and so forth.

Can you explain why this is unachievable other than by tagging your yield-capable call sites?

Re:Using Lua coroutines to create an RPG dialogue system by unscaled on Thursday January 12, 2017 @04:23AM (#13380949)
You can acheive some composability (e.g. easily awaiting multiple events) by having first-class values for the coroutines/green-threads themselves. This is what greenlet/gevent does, and it's rather pleasant to use. Try doing something like gevent.joinall() on Go though. https://nathanleclaire.com/blog/2014/02/15/how-to-wait-for-a...

Not fun.

The other problem is that even with gevent, what you get is a first-class _coroutine_, not a first class class _continuation_. You have to actually spawn new coroutines (creating new stacks and going through sometimes unnecessary scheduler hoops) to get a first-class coroutine out of a possibly blocking function.

Again, I'm not saying you can't do something like gevent.joinall(gevent.spawn(foo()), gevent.spawn(bar()), but:

1. Many languages don't support this (including Go and Lua, AFAIK). 2. It comes with its own tradeoffs again.

Re:Using Lua coroutines to create an RPG dialogue system by anonymoushn on Thursday January 12, 2017 @10:20AM (#13382845)
> 1. Many languages don't support this (including Go and Lua, AFAIK).

I don't think this is necessarily a property of the primitives. greenlet doesn't support anything like gevent.joinall(gevent.spawn(foo()), gevent.spawn(bar()), but gevent does by adding python code to greenlet. So I'm sure that in the same manner one could do it in Lua using Lua's coroutines. (There might be some concern that we have the wrong primitive: greenlet provides symmetric coroutines and Lua provides asymmetric coroutines. I don't think it matters for this which one we use, but if it does it is simple to implement either in terms of the other.)

That's not to say that the situation is ideal! To a great extent it's good if a language comes with a standard way to do this stuff! But it would be silly to say something like "Lua doesn't have inheritance" because it has a DIY inheritance construction kit instead of an "extends" keyword.

Re:Using Lua coroutines to create an RPG dialogue system by anonymoushn on Tuesday January 10, 2017 @10:08PM (#13371430)
The async/await model and the coroutine model are equally capable of expressing "start these 3 tasks and wake me up when they're all done." I've written this before in Python using Greenlet.
Re:Using Lua coroutines to create an RPG dialogue system by SomeCallMeTim on Wednesday January 11, 2017 @03:13PM (#13377116)
Lua coroutines don't have that ability out of the box.

JavaScript promises do (Promise.all).

Re:Using Lua coroutines to create an RPG dialogue system by anonymoushn on Wednesday January 11, 2017 @06:02PM (#13378269)
This is an interesting characterization of the situation! I can save the 20 LOC involved in implementing this if I just give up the ability to write my own event loops and pepper my code with a bunch of additional "await"s every time I add an asynchronous operation to a function used by other functions.
Re:Using Lua coroutines to create an RPG dialogue system by unscaled on Thursday January 12, 2017 @04:31AM (#13380985)
You can also implement your own event loops with promises, but it's a matter of tradeoffs again:

You can't control the way functions written by someone else will schedule their internal continuations - you can only control the way you schedule your own continuation. I'm not sure if Javascript gives you a lot of control there, but C# allows you to define your own TaskScheduler and specify it on ContinueWith.

Re:Using Lua coroutines to create an RPG dialogue system by pcwalton on Tuesday January 10, 2017 @03:32PM (#13368709)
> I think that, on balance, actually creating a data format that represents the state and "coding" using that plus state machines is actually a more robust and useful way to accomplish the same thing.

Manually coding up a "state machine" is kind of reinventing the Lua interpreter, no?

Re:Using Lua coroutines to create an RPG dialogue system by nine_k on Tuesday January 10, 2017 @03:45PM (#13368808)
Coding a state machine manually is indeed suboptimal.

Computers are good at generating code. If you want a very fast FSM, they will build it for you from a compact representation (cf. a regex).

But for game dialogs speed is probably a secondary concern. I suspect that a table-driven FSM interpreter with a fully transparent internal state that can be saved / loaded is not very hard to create. Your dialogs' input language becomes the control table, which is possibly no less designer-friendly than Lua coroutines.

Re:Using Lua coroutines to create an RPG dialogue system by optionalparens on Tuesday January 10, 2017 @07:59PM (#13370856)
Speed is not always a secondary concern for dialog. Certainly in many games where the dialog is very fixed, sure. However more and more, dialog systems actually need to be pretty fast. The simplest reason is you generally have only a few or even 1 frame to process dialog and you certainly don't want it tying up your frame's update cycle. It's easy enough most of the time as a place where you can shave off time with some good optimizations anyway for simpler cases.

An example I see a lot where it needs to happen fast is spatial queries or AI-related dialog. For instance if I walk by someone or something, that has to trigger a query on the dialog engine for someone to say something to the player. It's not just the dialog, but the sound assets, playback, etc. that need to happen and they need to happen in a way that isn't annoying - i.e. player is far away already before dialog starts or has to "wait" in place.

Sometimes as I mentioned, AI also gets thrown into the mix in different ways. The simplest is that you want some piece of dialog to be triggered by some other game states - ex: if player killed dragon, say this, otherwise say that. It seems simple, but when you start throwing in lots of world states like in open world games and want some decent variation and natural dialog that isn't awful and repetitive, it gets harder. Now add on spatial and other requirements for a dialog query and things start to suck. To this end, it's why a lot of dialog systems suck. It's not just laziness, rather sometimes people actually can't get their dialog systems to work fast enough during update cycles and end up dropping features because of the cost vs. benefit for most games.

Dialog seems simple, but is surprisingly complicated in the greater ecosystem of other game systems. But again, this depends on the game and if you just have a side-scroller where you repeat the same 5 catch phrases, then yeah, dialog is simple and speed is not a problem.

Re:Using Lua coroutines to create an RPG dialogue system by mmjaa on Tuesday January 10, 2017 @03:37PM (#13368738)
As a long-term Lua fan, I'm with you. I think that the OP is really complaining about not understanding the Lua VM well enough to push it to its limits - in fact, coding up a 'proper state machine' that can compete with what the Lua VM, itself, has to offer is going to be very, very difficult.

That said, I'm in favour of other approaches to this problem - using (i.e. generating) Lua bytecode itself as a way of maintaining state, and then pushing things around while having a good understanding of llimits.h seems to be quite important ..

Re:Using Lua coroutines to create an RPG dialogue system by SomeCallMeTim on Tuesday January 10, 2017 @05:01PM (#13369542)
Sorry, but you're pretty profoundly wrong.

I have written an entire (popular) game engine that tightly integrated Lua and used custom-generated C++ bindings. I had to dig pretty deep into the Lua VM to accomplish that.

Heck, I found and fixed a bug in the LuaJIT 1.x series -- my fix integrated by Mike Pall, with a credit on the changes page.

The thing is if you encode game state into the program state, like using coroutines for complex, long-running AI tasks, for instance, then there's no way to save or restore that state if you're using LuaJIT. And Pluto saves the state in an opaque manner.

And as another commenter pointed out, if you want to update the game with new behaviors, god knows how that will interact with save states that have huge call stacks in them. And bug fixes? Yeah, since you're saving the whole program state at this point, bug fixes won't be incorporated in the restored image.

Re:Using Lua coroutines to create an RPG dialogue system by optionalparens on Tuesday January 10, 2017 @07:49PM (#13370793)
Agreed. I've seen things like this tried in Smalltalk images and it was usually a disaster long-term, meaning that the state was simply thrown out and a new image spun up because you're SOL. Either that, or you need tons of logic to handle the old program state and it becomes a convoluted mess.

Most of the games and engines I've ever worked on that have dialog systems expressly avoid encoding things into the program state as much as reasonably possible. This is not only for restoring state, but also making for a saner experience editing and changing things during development. Very rarely does the format you encode things like dialog stay the same during a dev cycle unless you're using a canned engine and don't make any serious changes.

Using Lua program state itself seems like a way to drive yourself mad by the end of the dev cycle, and make it incredible unfriendly to work with writers and other content creators. People who work with asset pipelines often do a lot of cruel and unusual data conversions and munging to bridge how different people and systems need to work, but eventually things in the actual game engine are best kept as simple data that can be read in and reproduced at any time. Moreover, as a developer, I should be able to load up any state of my game at any point in the game just by providing the necessary input data as possible, and like a pure function, it should produce the same game state every time if possible. Let's also not forget other things like internationalization and localization.

As far as interfacing with Lua, I've mostly done it in the last few decades in a simple way where we handed entity component-style data that is just pure data, i.e. no function pointers and crazy stuff, and either let Lua do the rest with that state, or send pure data back to the C++ when needed. As for the data format itself, I've seen everything from XML to custom binary formats to CSV and Excel used for dialog. I think I'd sooner murder someone if game dialog lived in code, even in a Lua script.

Re:Using Lua coroutines to create an RPG dialogue system by davexunit on Tuesday January 10, 2017 @10:20PM (#13371472)
Data is code and code is data. There's really no difference.
Re:Using Lua coroutines to create an RPG dialogue system by optionalparens on Wednesday January 11, 2017 @08:46AM (#13373816)
No, this couldn't be further from the truth and is the reason why things like entity component systems exist - to graft data on to code and make data-driven programming easier. In Lisp-like languages, this is more true. In C++ which most games are written in and even Lua which is a common scripting language, this isn't really true. There is a wealth of information about how code is not data all over the web - google it please.

A few reasons why in C++, C, and Lua, and in general in most game programming languages code is not the same as data:

- Tends not to be editor friendly

- Requires heavy meta programming to achieve the same as other formats

- Requires compilation and/or interpretation

- Is not necessarily idempotent depending on compiler options and platform. The same data should be the same cross-platform, and code is almost definitely not because at a minimum level, it can generate vastly different assembly depending on compiler flags or processor targets.

- Purely composite. I would argue that as much as we try, it's hard to compose a lot of code properly and without considerable effort in games. Conversely, composing data can be quite easy.

- Cannot on its own be serialized/deserialized easy while in a runtime state

- Not network transparent

- Cannot be compressed/decompressed in real-time as easily

See my points in other posts in this thread. I am assuming you downvoted me, which is ridiculous and childish, especially given your brief response without any reasoning or evidence despite being contrary to all computer science research.

Re:Using Lua coroutines to create an RPG dialogue system by mmjaa on Thursday January 12, 2017 @06:33AM (#13381453)
Sorry, but as a long-time Lua user, I have not found any of your points to be true at all. Lua is truly editor friendly; I can even use cscope with it. Meta-programming is light, that is why it is meta-. Idempotency is one thing: shipping code is something else, and a well managed Lua project ships.

Composition: see meta-tables. Repeat until completion.

Re:Using Lua coroutines to create an RPG dialogue system by optionalparens on Thursday January 12, 2017 @07:15PM (#13387386)
Again, you're welcome to disagree. We're talking about game development in particular, not Lua in general. I've used Lua and almost always look to it for scripting in game engines I have built.

It's telling that all the major game engines and a large portion of papers and talks disagree with you. If Lua was the solution to data-driven development, people wouldn't bend over backwards looking for solutions like entity component systems. Lua is nice, but not suitable for most professional game development as a primary language. As such, it makes it completely unsuitable because Lua data, especially things that depend on runtime state are utterly useless.

Your shipping comment also makes no sense in this context. Tons of other languages have more games shipped than Lua. There aren't too many people relying on Lua for composition and data-driven development in games, because it doesn't work. Even Lisp doesn't work in this context putting speed and other things aside, because of so many reasons I've already listed. Code in the context of game development is not data by the definition of data I am referencing.

Re:Using Lua coroutines to create an RPG dialogue system by pcwalton on Tuesday January 10, 2017 @10:03PM (#13371404)
I'm not suggesting to try to persist coroutines—that is indeed brittle. Rather I'm suggesting that throwing away Lua in the process doesn't seem necessary.
Re:Using Lua coroutines to create an RPG dialogue system by SomeCallMeTim on Wednesday January 11, 2017 @03:30PM (#13377242)
I didn't throw away Lua because I couldn't persist coroutines. That's just why I didn't use Lua coroutines the way the article recommends; I had plenty of of other reasons to abandon Lua. [1]

[1] https://realmensch.org/2016/05/28/goodbye-lua/

Re:Using Lua coroutines to create an RPG dialogue system by anonymoushn on Wednesday January 11, 2017 @06:20PM (#13378409)
What do you recommend for programming the stuff that corresponds to this sort of state?

For a similar problem of making the AI code for a bullet hell shmup, I thought it was very important to be able to edit the AI code easily to try out a lot of stuff. I've had decent experiences using BulletML and Lua for this, but both of them provide no path forward if I want to update the game and load an old save. Like, because I was not asked to explicitly name every single state in my "AI program", I cannot create a mapping between the possible old states and the possible new states to use when running the updated software against an old save. Then again, the fact that I was not asked to explicitly name every single state in my program feels like a pretty central component of the short iteration time I was after to begin with.

For shmup AI this maybe ends up being acceptable because the game doesn't need to run new behaviors against old save files. But if I did need that, maybe I should be using two systems, one for experimentation and another one for making something upgradable?

Re:Using Lua coroutines to create an RPG dialogue system by mmjaa on Wednesday January 11, 2017 @03:29AM (#13372568)
Through the proper use of byte code generators none of your straw man arguments are relevant. (BTW, I've also built a game engine using Lua, and shipped many titles with it, so ..)
Re:Using Lua coroutines to create an RPG dialogue system by SomeCallMeTim on Wednesday January 11, 2017 @03:40PM (#13377314)
The Playground SDK had over a hundred titles shipped on it. You can see some of them on my MobyGames profile. [1]

"Proper use of byte code generators" -- at that point you're not writing Lua any more, any more than Scala or Clojure targeting the JVM is "writing Java." The article is about using Lua for AI, which is what my comments are about, and my comments all stand.

Arguing about whether the VM is good enough to support a state machine is orthogonal to whether it's a good idea to use Lua coroutines as suggested by the article.

[1] http://www.mobygames.com/developer/sheet/view/developerId,13...