News for hackers. Stuff that matters.
Re:Databases: 2021 in Review and Predictions for 2022 by stevefan1999 on Friday November 11, 2022 @06:32AM (#33559454)
The very problem of SQLite: Single user only. Although SQLite does have WAL but it still doesn't allow you to do concurrent write unless you want to see file corruption.

This means SQLite is very much locked to things that works with one specific purpose and almost nothing else. Sure, you can be read-only, but you have to run alongside the app in the specific node, too.

Another problem (although without solving the single user mindset this wouldn't be a problem at all) is high availability. You want to make sure that your database won't get lost do you.

Things like Litestream [1] attempts to solve the SQLite backup problem it by continuously saving the database state and pack t up to S3-compatibles or file system but its just half the story. You want to make sure your operation not stopping. This is where HA comes in to save you from an emergency fixup when you are enjoying your holiday.

It doesn't mean that nobody tried to solve both these problems though. Ahem, introducing rqlite [2]. Although my own experience is not very great because the memory usage is quite high and does not fit my need (because the embedded device only has 512MB on it, and every byte counts, sorry), I guess that's the price to pay if you want to turn a non-multiuser, non-concurrently acccess database into one...Another honorable mention would be LiteFS [3] but I haven't used it yet so I have no say on it.

[1]: https://litestream.io/

[2]: https://github.com/rqlite/rqlite

[3]: https://github.com/superfly/litefs

Re:Databases: 2021 in Review and Predictions for 2022 by thunderbong on Friday November 11, 2022 @06:43AM (#33559523)
Many of us here writing web apps for enterprises can use SQLite with WAL with no issues.

Number of concurrent users range from tens and rarely hit hundreds. SQLite can handle that kind of traffic without any issues.

Re:Databases: 2021 in Review and Predictions for 2022 by mritchie712 on Friday November 11, 2022 @06:53AM (#33559616)
why would you do all this work when postgres sorted this all out over a decade ago?
Re:Databases: 2021 in Review and Predictions for 2022 by collyw on Friday November 11, 2022 @07:22AM (#33559907)
Setting up Postgres is a PITA compared to SQLite. It comes bundled with python these days. Obviously it's going to be a trade off as to which one causes you more pain.
Re:Databases: 2021 in Review and Predictions for 2022 by heurisko on Friday November 11, 2022 @07:37AM (#33560067)
No docker access? postgres images are available in seconds.
Re:Databases: 2021 in Review and Predictions for 2022 by tehbeard on Friday November 11, 2022 @08:35AM (#33560634)
So the solution is pulling down additional gigabytes of images and runtime to run the database?
Re:Databases: 2021 in Review and Predictions for 2022 by counttheforks on Friday November 11, 2022 @09:04AM (#33560900)
Yes. It works great.
Re:Databases: 2021 in Review and Predictions for 2022 by noAnswer on Friday November 11, 2022 @12:29PM (#33563764)
I manly work as a sysadmin for small companies. (Most people don't call us when they install new tools, they call when they exploded.) All my hatred goes out to the (Windows) programs (and its creators) that think they need MS SQL Express or the likes to save their two bytes of dust. All my love goes to the programs that just run/save from/to a UNC path.
Re:Databases: 2021 in Review and Predictions for 2022 by TristanBall on Friday November 11, 2022 @02:49PM (#33565845)
Has SMB locking, and OP Locking become that much more reliable?

Pretty much one of the classic desktop support calls for me was "my access database on the shared drive is corrupt"

I'm out touch now, but one of the failure modes seemed to be that a client would take out an oplock, so it could do local caching etc. Then someone else opens the file, the server sends the oplock break to original client but that message gets lost/ignored, and we end up with 2 or more clients now making unsynchronised changes to a file.

Any smb client access to shared data more complicated than documents and spreadsheets just makes me twitch these days.

It was a long time ago now, and it probably was more prevalent in larger environments.. just more chances for things to go wrong I'd guess.

If something is designed for shared filesystems that's different, but my experience was that at the low midrange things aren't. They seem to work, until they don't.

What's wrong with sql express? Assuming you fit within its size constraints?

Re:Databases: 2021 in Review and Predictions for 2022 by noAnswer on Friday November 11, 2022 @04:57PM (#33567440)
> Pretty much one of the classic desktop support calls for me was "my access database on the shared drive is corrupt"

Yes, that one >12 year old access database with two concurrent users brakes biweekly. Everything else: No problems.

Just this Thursday I "moved" a program from an old PC to a new one. We more or less support this client since 7 years. The PC is from before that. Got the call. First time hearing about that lab software. Right click on the desktop icon. Oh, it just points to an UNC path on the "main server" (Linux Samba share). Took a look into the settings. It has a path to a SQLite DB. Also just an UNC path. Only two clients using it. It worked unattended for years. Since it's stored on the "main server" it's also covered by backup. No client side installation necessary. Copy the link. Update our documentation that we know it exists, what is it used for and how it works. Done.

I see it time and time again: Desktop software that runs on one PC, produces/saves tiny amounts of data (like a temperature monitor for one fridge), but installs a SQL server. A random technician shows up, installs it and leaves. We don't know that it exists. It's therefore not part of a backup concept. It's usually only discovered when it explodes or when the PC is renewed (or when a second PostgreSQL install creates a TCP port conflict). I have the feeling only 1 out of 10 "softwares" even have a proper concept for export and import of data.

A client with like 7 or 8 workers called my because he wanted me to move his new time tracking software. The technician who also installed the NFC reader apparently installed it on the wrong PC. It took over 3 hours! It downloaded GBs and GBs of .NET, SQL Express and what not. I didn't know that going in/clicking the setup.exe. I had to call support to get the Express DB over. I can still hear the hard drive screaming. ...man... for stuff a Z80 and a CSV file would be enough. ...And I am the bad one for explaining to the client that the DB needs to be backuped. I sound like I want to upsell him something.

I know, I know. This rant is not about databases. It's about the state of IT and arrogant developers who don't live in the trenches. The "pulling down additional gigabytes of images and runtime to run the database" > "Yes. It works great." triggered me. I'm sorry. You know. Not everything is a unicorn webapp.

> Any smb client access to shared data more complicated than documents and spreadsheets just makes me twitch these days.

I know of multiple large install bases of an old medical software. It's still actively developed/supported but it is so old that it doesn't use a SQL but ISAM database. It just lies there, on a file share. No active server component to speak of. No client site installation necessary.

> What's wrong with sql express? Assuming you fit within its size constraints?

If it is not used for something a CSV file would be enough. If the costumer is told that it needs to be installed "on a server". If we get involved from the beginning so we know what is what. If the costumer doesn't expect me to be able to move it out of the blue within seconds. Than sql express is fine. ...which reminds me of that one time, when that well-hung developer of a software for the energy sector didn't knew that express only can allocate up to 2GB. Obviously he was blamed us for delivering a faulty server...

Re:Databases: 2021 in Review and Predictions for 2022 by heurisko on Saturday November 12, 2022 @02:01PM (#33575951)
postgres alpine is 90MB.

I can really recommend it, using a declarative docker-compose.yml file and then docker-compose command.

Re:Databases: 2021 in Review and Predictions for 2022 by mritchie712 on Friday November 11, 2022 @07:34AM (#33560032)
agreed, running pg locally is a pain. I use a cloud postgres instance (even for local dev). They're dirt cheap and it's not worth the hassle of working with a local pg.
Re:Databases: 2021 in Review and Predictions for 2022 by kikowi on Friday November 11, 2022 @07:37AM (#33560069)
which cloud do you use for local dev?
Re:Databases: 2021 in Review and Predictions for 2022 by mritchie712 on Friday November 11, 2022 @07:41AM (#33560100)
aws, gcp, digital ocean and supabase all have pretty good free tiers
Re:Databases: 2021 in Review and Predictions for 2022 by jacquesc on Friday November 11, 2022 @08:05AM (#33560344)
I love Postgres.app on mac, makes running locally a breeze. Running Postgres fast on the cloud is what I always struggled with (without paying enormous sums)
Re:Databases: 2021 in Review and Predictions for 2022 by counttheforks on Friday November 11, 2022 @09:05AM (#33560909)
What OS are you on? Running postgres on Linux is as trivial as it gets, and for Windows they have a nice installer. Or just use docker.
Re:Databases: 2021 in Review and Predictions for 2022 by collyw on Monday November 14, 2022 @06:31AM (#33592631)
Does the docker have users set up? I seem to remember that being a pain point in the initial setup.
Re:Databases: 2021 in Review and Predictions for 2022 by osigurdson on Friday November 11, 2022 @04:16PM (#33566970)
>> agreed, running pg locally is a pain

If you have a single process accessing the database, use SQLite. If not, use Postgres.

Re:Databases: 2021 in Review and Predictions for 2022 by unemployable on Friday November 11, 2022 @07:54AM (#33560221)
There is also Cloudflare D1, which is now in public alpha.