Yeah ... but nah.

Yeah, there should be some kind of shakedown test.

But Nah, not every issue can be tested for within the manpower/$ budget. Issues of load being one of them.

A valid strategy is: send it!, work with the beast a bit, if too unruly roll it back. In this case, they tamed the new beast. Not like Lichess is going to lose billions...

Load testing isn't quite the expensive proposition it seems!

Think about it - clients or requests are really just ephemeral ports...

There are something like 30 thousand of them available by default with Linux

If you use sockets you can have as many connections as your system allows for open file descriptors. The upward limit is...

    $ sysctl fs.file-max
    fs.file-max = 9223372036854775807

Spawn a ton of concurrent requests to hit the service/API! apachebench is a decent example of the idea - imagine some proprietary sauce with it

It gets cheaper (and in some ways, more effective) if you actually remove the network -- latency goes down

You can also rate limit it in software, but it's really not analogous in some ways (eg: net.core.somaxconn limitations/realities)

Development time is totally understandable though. It's important I leave my green field of comfort

Load testing would have to be quite advanced to find this, doing more than just simulating users for the most common features. It took more than 24 hours of real-world usage to visit sufficiently many niche code paths to fill the code cache.

Advanced in that it's POST instead of GET, and some parameters to boot

I'm not saying it doesn't take effort, I'm just saying it can and is regularly done

Regression testing with speed. If you test your features as they're added, niche-ness doesn't matter - you know about it and test it

A regression test quickly turns into a load test if you do them all at once

To be clear, I'm not trying to twist the knife. I understand why it wouldn't be done. I just won't pretend it can't

I've been trying to avoid this 'hindsight is 20/20' thing...

I'm not trying to lay blame - just offer thoughts towards improvement, and encouragement against some defeatist thinking patterns

A lot can be done with very little equipment/time investment

Lichess being a project that is 85% written by one guy, and all other changes approved and read by him, he may have felt confident (until now) that he could maintain all the complexity of the software in his own head, and recall what to do and where to do it whenever he made changes. Seems bonkers to most of us writing software in teams, but compare this choice of workflow to ones made by others in similar situations: programmers in academia, who often work entirely alone, or Linus Torvalds in earlier versions of the kernel. Skipping testing is a common choice, especially when working alone and with a typed language. I could see Thibault bringing tests in now however. They would not have caught much in this case, because it seems like the problem came to light only when the software started running with real users.

While I do agree, these folks deserve a break... see my other comment in this thread RE: load testing

It's not particularly expensive or difficult, but it does need investment