It takes a very dedicated plan to sufficiently test apps at scale when we're talking the scale of 300 million or billions... In this case the scale was 1700 (maybe off by an order of magnitude) when your scale is on the thousands or tens of thousands you can't do outright stupid stuff... but you can do reasonably dumb stuff and be fine - running a test with a hundred or so users should've clearly demonstrated these issues and there are companies out there that excel at resourcing a few minutes of a few hundred folks' time for just this kind of load testing.

That said, this was a pretty pathetic failure to scale, and I don't use harsh language like that lightly. I assume we're going to find out someone's brother or nephew runs the company because, for whatever reason, politics loves nepotism.

This is a small scale app by any sensible standards and probably not that complex compared to what's being routinely deployed by 2-3 man development teams in the world of business. i routinely handle spreadsheets with larger datasets and more complexity then whats involved here.

There is books written on how to test an new system/feature but in general you have 3 stages:

The first is an automated internal integrity test(sometimes called unit test) where you throw mock data at the systems independent component, in order to test for known classes of edge cases and race conditions.

The second step is implementation where the ops team tries to yank network and hardware resources away form underneath the app until something breaks in order to see if it fails in a sensible way that don't lead to irrecoverable data corruption.

The third stage is acceptance testing which on proprietary solutions is often the only one the end user is 110% in control of where you both test the app with actual user performing standard procedures and an team of specialist trying and break it using whatever knowledge they have on the systems design.

All tree stages have it's own skill set and is often performed by different teams and it's not uncommon to see specialist companies brought in doing part of the QA process for really important new systems but for the most part the entire things just fade into the the daily routine as an newer ending feedback loop.

This isn't really "scale."

It needs to record something like a dozen numbers from each of the ~1,700 caucuses in the state. You could have used Excel as the database here with no problem.

I was in the shower this morning thinking about how I would solve this, and something like this is exactly what I would have done. Use simple, familiar, battle tested technology to do the heavy lifting.

On production, apparently.

Iowa has 1,681 precincts. This is not a scalability problem.

Even the simplest CRUDs will have problems when you have 1681 users who must use it and you can't just bounce out the ones who don't get it and chalk it up to bad user fit.

If your CRUD can't handle 1681 users, then you're almost certainly doing something horribly, horribly wrong.

Like, that's "LAMP stack in 2001 hosting someone's blog about cacti wearing sweaters in Duluth" levels of scale.

"Scale" in this case is 1,681 precincts, each with a handful of users (the election officials). I'm not sure that was the issue in this case.

There are apps and libraries that can simulate load.