I recently had Jon Calhoun on as a guest in my Running in Production podcast the other week where he talked about using SQLite to power his video course platform (it's focused on Go development).
He has 15,000+ active users and talks about how SQLlite hasn't been a performance issue. His whole stack is on a $5 / month DO server.
Here's a direct link to the SQLite part: https://runninginproduction.com/podcast/42-creating-a-video-...
I also have another episode coming out next week where SQLite is being used in a custom hardware appliance running on a Raspberry Pi-like device that is shipped to villages in the African Congo to give villagers affordable access to email. I wonder if the creator of SQLite had that use case in mind!
Just because it's not used in web applications as much doesn't mean it's not production ready. It's used in many hardened systems, which I think they'd avoid if it wasn't production ready.
Also HN: "Database X can fail in Jepsen tests during partitions, it's unusable"
I see HA as a mandatory part of what makes a system production ready...
Most websites are not Google or Amazon, and gain very little from more 9's past the initial 2~3, which is totally manageable on a single server. Why spend thousands setting up and managing distributed servers if going from 99.0% to 99.999% gains you hundreds? If you need more nines and stronger guarantees than that, absolutely, use distributed solutions, nobody is telling you otherwise.
Jepsen tests are valuable once you have decided to use a distributed solution. They showcase problems that would be extremely hard to track down in production settings but can cause serious problems there. If encountered in practice these problems can cause data corruption or other very strange bugs that would take days if not weeks to track down.
I wouldn't say that a system with failures in Jepsen tests is unusable, every system has bugs, but if a system has these problems on purpose (e.g. to make it look better on benchmarks) and there is no movement to fix these problems then I would definitely steer clear of that system.
"How much money?" Is probably the most import question, as it actually helps in evaluating if there's a business case for investing into the financial commitment, engineering effort and maintenance burden of an ha solution.
On a different note, I've seen mysql-based multi-master setups and it's really a joy to be able to treat a mysql crashing as a minor annoyance instead of a call to fire-fighting.
Edit: there is also more to be said about this. I'd research crawl-budget if you're interested.
It's likely the additional overhead worrying about HA is not worth it, not to mention the real possibility of HA just not working properly in actual failure scenarios
It seems like his business could at least afford an HA setup on a cloud provider. Moving to any hosted db with backups, updates and redundancy could be worth it.
As for HA failing, it's still less likely than not-HA failing.
Also an HA setup allows for maintenance and upgrades without downtime, which is way better than the very common "we don't upgrade if it's working, because it could break".
Also there is a third option between HA (with its increased cost and complexity) and "we don't upgrade if it's working, because it could break", which is "take the site down for a few minutes, do the upgrade, bring it back up". It's not for every site, but there's a range of sites for which that is fine.
I'm using sqlite for a production application as well, but its load will be in the range of - at best - thousands per day. The only complexity will be that there has to be a secondary 'fallback' server; the current application just copies the whole .db file over to the secondary server after certain events.
FWIW the typican front-page article in HN sees around 10-30k visits in a single day.
If you can somehow architect things so a single process is doing writes (but leave multiple processes doing reads) that's likely even better.
However I would suggest doing a lot of testing. It's possible multiple write processes isn't the end of the world below some threshold.
Thanks
From the website [1]
> SQLite is used by literally millions of applications with literally billions and billions of deployments.
It's also worth noting (at least, imo) that it's automated test suite is fantastic. I seems to recall it has considerably more test code than actual library code.
> Rails comes with built-in support for SQLite, which is a lightweight serverless database application. While a busy production environment may overload SQLite, it works well for development and testing. Rails defaults to using a SQLite database when creating a new project, but you can always change it later.
> By default, the configuration uses SQLite. If you’re new to databases, or you’re just interested in trying Django, this is the easiest choice. SQLite is included in Python, so you won’t need to install anything else to support your database. When starting your first real project, however, you may want to use a more scalable database like PostgreSQL, to avoid database-switching headaches down the road.
In fact, I would argue SQLite is the most well-tested and production ready database system out there.
I would doubt that is what the creators of SQlite intend one to take away from their documentation:
"SQLite works great as the database engine for most low to medium traffic websites (which is to say, most websites). The amount of web traffic that SQLite can handle depends on how heavily the website uses its database. Generally speaking, any site that gets fewer than 100K hits/day should work fine with SQLite. The 100K hits/day figure is a conservative estimate, not a hard upper bound. SQLite has been demonstrated to work with 10 times that amount of traffic.
> The SQLite website (https://www.sqlite.org/) uses SQLite > itself, of course, and as of this writing (2015) it handles > about 400K to 500K HTTP requests per day, about 15-20% of > which are dynamic pages touching the database. Dynamic > content uses about 200 SQL statements per webpage. This > setup runs on a single VM that shares a physical server > with 23 others and yet still keeps the load average below > 0.1 most of the time." (Source: https://www.sqlite.org/whentouse.html)
So SQlite is quite a valid choice if you need a database and stay below ~500k HTTP requests per day. So it probably is "production ready" for many a site.