News for hackers. Stuff that matters.
Re:Why you should probably be using SQLite by filleokus on Friday October 27, 2023 @09:15AM (#38038025)
Yeah. I've gotten into discussions on HN before about this. sqlite is awesome when you run it in one instance on a VM or something, but solving migration between hosts or backup via the filesystem seems like a mess. Sure LiteFS helps, but is it better than a managed networked database?

> One huge benefit to SQLite is the fact that it runs as an embedded part of your application

But all production workloads I've been involved with the last ≈5 years have either been containerised apps offloading state to databases/caches/blob storage etc, or have strived towards that.

What I do think would be awesome would be an embeddable Postgres library/binary that could use a single state file on your local filesystem for development and use a networked database in production. Getting the benefits I like most about sqlite locally, and not having to deal with files in production.

Re:Why you should probably be using SQLite by Aurornis on Friday October 27, 2023 @09:26AM (#38038149)
> sqlite is awesome when you run it in one instance on a VM or something,

This article really needs to be taken in the context of the author’s target audience: He sells courses that primarily target web dev juniors doing learning projects and small projects. That’s why he includes the “most of you reading this” disclaimer in the article:

> So, can you use SQLite? For the vast majority of you reading this, the answer is “yes.” Should you use SQLite? I’d say that still for the majority of you reading this, the answer is also “yes.”

Pushing strong contrarian opinions is part of his social media marketing style. He’s also pushing a “why I won’t use next.js” article and opinion on his social media, while conveniently omitting the obvious conflict of interest that he sells courses that don’t happen to cover Next.js.

Re:Why you should probably be using SQLite by thaumasiotes on Friday October 27, 2023 @09:49AM (#38038393)
By the time you're labeling stating your opinions in public and then acting consistently with them as a "conflict of interest", you've abandoned any ability to get anything useful out of the term.
Re:Why you should probably be using SQLite by szundi on Friday October 27, 2023 @10:11AM (#38038657)
My thoughts exactly
Re:Why you should probably be using SQLite by yawaramin on Friday October 27, 2023 @10:33AM (#38038970)
So if I like apples more than oranges, I write a blog post saying I like apples and won't eat oranges, and I also have a business selling apples...it's a conflict of interest?
Re:Why you should probably be using SQLite by Aurornis on Friday October 27, 2023 @10:49AM (#38039220)
If a company sells apples and they start pushing social media campaigns about why oranges are bad and they think you should be buying apples instead, then that’s a conflict of interest.

I think some people are missing the conflict of interest because the author blurs the lines between his personal opinion and his business. He turned his personal brand into his business and uses his name as his brand.

Re:Why you should probably be using SQLite by thaumasiotes on Friday October 27, 2023 @12:10PM (#38040377)
> I think some people are missing the conflict of interest because the author blurs the lines between his personal opinion and his business.

No, what's going on is that you don't understand what "conflict of interest" refers to.

Different people having different interests, as when the apple company says you should buy apples even though you prefer oranges, is just a regular conflict. A conflict of interest is when one person has two different interests. The apple company isn't experiencing a conflict in your example.

Re:Why you should probably be using SQLite by Sohcahtoa82 on Friday October 27, 2023 @12:31PM (#38040626)
> If a company sells apples and they start pushing social media campaigns about why oranges are bad and they think you should be buying apples instead, then that’s a conflict of interest.

Pretty sure that's just marketing.

A lot of product pages will spell out features they have and list competitors that don't have those features.

Re:Why you should probably be using SQLite by brvsft on Friday October 27, 2023 @12:38PM (#38040705)
We're missing the "conflict of interest" because you are abusing the meaning of the term.
Re:Why you should probably be using SQLite by albedoa on Friday October 27, 2023 @02:45PM (#38042333)
> e.g., that Nike has a conflict of interest because it is their opinion that you should purchase their shoes.

It's even worse: that Nike has a conflict of interest because they think that wearing shoes while running is good, even though it would be extremely weird if a running shoe company thought that running with shoes is bad.

Remove the business from the equation and the opinion is fine — it's the existence of a business that aligns with the opinion that creates the conflict! Galaxy brain definition.

Re:Why you should probably be using SQLite by rrrrrrrrrrrryan on Friday October 27, 2023 @01:35PM (#38041454)
> EDIT: I think people are missing the meaning of "conflict of interest". Having a conflict of interest doesn't mean something is wrong. You can be right and have a conflict of interest.

He has motivations that make him not impartial, but that's not usually termed a "conflict of interest."

You're just using the term strangely. Usually the phrase "conflict of interest" implies the person has some sort of official commitment / obligation / duty to another party that is put in jeopardy because of another conflicting interest.

But this is just some guy selling stuff. He hasn't made any official commitments; he doesn't have any official duty or obligation to remain impartial.

By your definition, anyone selling anything has a conflict of interest because they want money from their customers, which may be against the customers' best interests.

Re:Why you should probably be using SQLite by jimbokun on Friday October 27, 2023 @11:05AM (#38039442)
If you don't disclose the business, yes.
Re:Why you should probably be using SQLite by willsmith72 on Friday October 27, 2023 @11:13AM (#38039549)
It's extremely disclosed
Re:Why you should probably be using SQLite by rmbyrro on Friday October 27, 2023 @10:37AM (#38039026)
Since he doesn't like next.js, you expected him to offer a course on it?

I really don't get the reasoning here...

Re:Why you should probably be using SQLite by Aurornis on Friday October 27, 2023 @10:52AM (#38039262)
He made his courses before Next.js rose to popularity.

I wouldn’t be surprised if he offered Next.js courses if it becomes popular enough in the future. For now, the anti-Nextjs push seems like a clear defensive play to steer people back to the course material he already had for sale.

Re:Why you should probably be using SQLite by willsmith72 on Friday October 27, 2023 @11:12AM (#38039534)
That's just utterly untrue.

The whole course he's been advertising is based on remix, which came up YEARS after next.js rose to popularity.

Where did you get this info that you were so confident on?

Re:Why you should probably be using SQLite by no_wizard on Friday October 27, 2023 @11:44AM (#38040035)
He’s also employed as part of the Remix team, that’s part of it too
Re:Why you should probably be using SQLite by adamesque on Friday October 27, 2023 @12:12PM (#38040401)
He used to be but I think he left to do this EpicWeb thing.
Re:Why you should probably be using SQLite by flanbiscuit on Friday October 27, 2023 @01:12PM (#38041135)
Even with his bias you still saw a lot of agreement to his criticisms on the NextJS subreddit (which you would assume was biased towards NextJS) and the React subreddit.

https://www.reddit.com/r/nextjs/comments/17giozq/why_i_wont_...

https://www.reddit.com/r/reactjs/comments/17gl76d/why_i_wont...

Re:Why you should probably be using SQLite by leerob on Friday October 27, 2023 @01:14PM (#38041159)
I'll be posting a full response to this, which I think you'll find very interesting (from the opposite point of view). Stay tuned.
Re:Why you should probably be using SQLite by flanbiscuit on Friday October 27, 2023 @04:36PM (#38043582)
looking forward it!

Last time I truly used NextJS was in 2020 (it was v9), and that was only to make a statically generated brochure type site. I had started the site with Gatsby and didn't love Gatsby, so I switched to NextJS and loved it.

Recently I did start up a Next v13 project using the new App Router. But I haven't gotten that far past the initial scaffold and creating a couple components. Seeing all of these complaints about v13, especially the App Router, are making me want to scrap that and use something else.

Re:Why you should probably be using SQLite by leerob on Friday October 27, 2023 @07:38PM (#38045352)
Re:Why you should probably be using SQLite by password4321 on Friday October 27, 2023 @02:11PM (#38041914)
Discussion seemed most opposed to Vercel-ification and churn.

https://news.ycombinator.com/item?id=38018217

Re:Why you should probably be using SQLite by appplication on Friday October 27, 2023 @09:37AM (#38038272)
> What I do think would be awesome would be an embeddable Postgres library/binary that could use a single state file on your local filesystem for development and use a networked database in production.

I mean this is more of less what using a local container for Postgres is. It’s not exactly one file but it’s hidden in a volume somewhere where you don’t need to go file digging, so for all intents and purposes sure, it’s one file.

Most ORMs also will let you swap out your RDBMS so you can run sqlite locally and Postgres in prod via the same code. I think this is a bit of an anti pattern though. While the two are similar in most standard use cases, it’s not always apparent when you veer off that standard use case and into something that’s not supported by sqlite. And then all of the sudden you either have a divergence between what works locally vs in production or you start to intentionally limit your app’s capability to only what works with sqlite because you can’t locally validate it.

Re:Why you should probably be using SQLite by geraldwhen on Friday October 27, 2023 @10:44AM (#38039120)
You will run into compatibility issues almost immediately. Running a local Postgres container is really the only option.
Re:Why you should probably be using SQLite by time0ut on Friday October 27, 2023 @11:06AM (#38039452)
My general pattern these days is a compose file with postgres and another container to run my migrations (along with anything else I need). Then I have an integration test suite that hits all the code paths that touch the database. The whole workflow gets run again in CI but with the new image via a profile in my compose file. Its so fast and easy to do at this point.
Re:Why you should probably be using SQLite by captn3m0 on Friday October 27, 2023 @12:09PM (#38040368)
Managing Postgres, even via containers is not so simple. Upgrades between major versions will still need a backup+restore for eg.

I run a single container Postgres for my homeserver, and have to deal with occasional issues that wouldn’t apply to SQLite (which I run as well).

Re:Why you should probably be using SQLite by thaumasiotes on Friday October 27, 2023 @09:54AM (#38038438)
> but solving migration between hosts or backup via the filesystem seems like a mess.

I don't understand this. How would you solve backups without using the filesystem?

Re:Why you should probably be using SQLite by casselc on Friday October 27, 2023 @10:36AM (#38039019)
Re:Why you should probably be using SQLite by thaumasiotes on Friday October 27, 2023 @07:21PM (#38045231)
Why is this considered "not using the filesystem"?

What's the conceptual difference between backing up your Oracle database to an Oracle Cloud Infrastructure bucket and backing up your sqlite database to an S3 bucket?

Re:Why you should probably be using SQLite by filleokus on Sunday October 29, 2023 @04:00PM (#38062116)
I mean the filesystem is going to be used somewhere of course, and everything depends on your setup / the scale of things.

My point was more that with a networked database, you can easily centralise your backing up of databases. If you are in the cloud with managed databases, the backing up is kinda solved out of the box. If you are doing it yourself or on-prem, you probably have a database cluster of some kind where backing up can be managed (which of course involves a filesystem at some point).

But if you are using sqlite, every sqlite file across your organisation needs to be backed up. You'll probably have some scripts creating the actual snapshot and then some other scripts shipping that away somewhere. Those scripts probably need to be quite bespoke for different hosting solutions. Are you going to ssh into the vm's running the apps and do the backup that way, or will you be running the backing up scripts on each vm?

Of course if you have a single VM hosting a few apps with one sqlite database each, none of this is an issue, and it might be easier in the short term than a networked database. Or even if you are operating at "scale" and treating your VM's more like pets than cattle, it's probably not going to be an issue either.

Re:Why you should probably be using SQLite by eduction on Friday October 27, 2023 @10:39AM (#38039073)
> What I do think would be awesome would be an embeddable Postgres library/binary that could use a single state file on your local filesystem for development and use a networked database in production.

Absolutely! And also, short of that but incrementally building toward it, some sort of simplification of initial db setup and configuration, including really solid, thoughtful defaults (many of them are but I am guessing they could be substantially improved).

I feel like half of people’s attraction to SQLite is the easy setup and first deploy story. Postgres will never match that but could get closer.

Re:Why you should probably be using SQLite by jimbokun on Friday October 27, 2023 @11:07AM (#38039464)
> I feel like half of people’s attraction to SQLite is the easy setup and first deploy story.

That is an incredibly important part of the story!

Spending a lot of time on configuring and monitoring and operationalizing your database can kill a business iterating on its minimum viable project.

Re:Why you should probably be using SQLite by icedchai on Friday October 27, 2023 @12:36PM (#38040690)
In production, use a managed service (RDS, Cloud SQL, tons of others...) In dev, set up a Docker container. You won't spend a lot of time on either of these things.
Re:Why you should probably be using SQLite by never_inline on Friday October 27, 2023 @11:28AM (#38039773)
You can get very close to this with docker compose.

You can delete and recreate the volume when you want with `docker compose down --volumes` flag, which is great for testing.

Shameless plug: https://mahesh-hegde.github.io/posts/postgres_local_using_do...

Re:Why you should probably be using SQLite by sa46 on Friday October 27, 2023 @12:56PM (#38040943)
> What I do think would be awesome would be an embeddable Postgres library/binary that could use a single state file on your local filesystem for development

I wired Postgres up for our local dev. I don't believe in mocking the database, so all our tests and local dev run against a real Postgres instance.

The main tricks:

- Store the Postgres installation in a known location.

- Set the dynamic loader path to the installation lib dir, e.g., LD_PRELOAD.

- Don't run CI as root (or patch out Postgres' check for root)

- Create and cache the data directory based on the source code migrations.

- Use clone file to duplicate the cached data directory to give to individual tests.

One thing I'd like to pursue is to store the Postgres data dir in SQLite [1]. Then, I can reset the "file system" using SQL after each test instead of copying the entire datadir.

[1]: https://github.com/guardianproject/libsqlfs

Re:Why you should probably be using SQLite by thom on Friday October 27, 2023 @02:01PM (#38041794)
Little use if you’re not on the JVM but I’ve had great success with Embedded Postgres:

https://github.com/zonkyio/embedded-postgres

Each test just copies a template database so it’s ultra fast and avoids the need for complicated reset logic.

Re:Why you should probably be using SQLite by sa46 on Saturday October 28, 2023 @12:05AM (#38047039)
Yep, I used that trick as well. The evolution we went through:

1. Each test runs all migrations on a fresh database. Each test spends 2.1 seconds on db setup.

2. Each test suite runs all migrations. Each test copies the template database from the test suite. Each suite takes 2.1 seconds to run migrations, but cloning a template database takes 300 ms.

3. Bazel caches the data dir and rebuilds it once for all test suites. Reduces the initial test suite setup from 2.1 seconds to 160 ms (to copy the datadir).

4. Each test in a suite uses clonefile to copy the datadir. Reduces db setup overhead from 300 ms (to copy a template database) to 80 ms (to clonefile a datadir).

Currently, most of our testing overhead is clonefile and cleaning up the datadir. I'm interested in a single file sqlite FS because I could be clever and use LD_PRELOAD to replace Postgres's file system operations with sqlite, avoiding most syscalls altogether.

Re:Why you should probably be using SQLite by thom on Saturday October 28, 2023 @04:27AM (#38048164)
Nice, any chance you’ll package all this up one day? I appear to be getting better numbers than these right now but I suspect it’s because the DB is simpler and fairly empty to begin with, so presumably we could be going even faster.
Re:Why you should probably be using SQLite by loloquwowndueo on Friday October 27, 2023 @02:47PM (#38042349)
Point Postgres data directory to a memory-backed tmpfs, makes the thing blazing fast.
Re:Why you should probably be using SQLite by paulryanrogers on Friday October 27, 2023 @11:30PM (#38046860)
I've done this for MySQL too. Saved our CI times on GitHub Actions after upgrading to MySQL 8 and the single threaded performance dropped.
Re:Why you should probably be using SQLite by sa46 on Friday October 27, 2023 @11:53PM (#38046983)
I've tried--macOS doesn't make it easy. It's not a clear performance win since tmpfs with Postgres fsync=off is mostly memory-backed storage via the page cache already.

Another problem is that parallel tests can exhaust memory.

The slowest part of our tests is syscall overhead. A mostly empty Postgres data dir for a medium-sized database with a few hundred tables consists of 3k files. On my M1 macOS, it takes 120 ms to delete the entire data dir. Copying is cheaper at 80 ms using clonefile.