News for hackers. Stuff that matters.
Re:The Great Database Migration by pocketarc on Sunday June 30, 2024 @02:43PM (#40839128)
> Furthermore, bundling an 80MB+ SQLite file to our codebase slowed down the entire Github repository and hindered us from considering more robust hosting platforms.

It's... an 80MB database. It couldn't be smaller. There are local apps that have DBs bigger than that. There is no scale issue here.

And... it's committed to GitHub instead of just living somewhere. And they switched to Neon.

To me, this screams "we don't know backend and we refuse to learn".

To their credit, I will say this: They clearly were in a situation like: "we have no backend, we have nowhere to store a DB, but we need to store this data, what do we do?" and someone came up with "store it in git and that way it's deployed and available to the app". That's... clever. Even if terrible.

Re:The Great Database Migration by BHSPitMonkey on Sunday June 30, 2024 @04:56PM (#40840303)
> It's... an 80MB database. It couldn't be smaller. There are local apps that have DBs bigger than that. There is no scale issue here.

It depends. If that 80MB binary file in git is updated/replaced often, you likely have a problem (every 100 changes/replacements might grow the repo by as much as 8GB).

Re:The Great Database Migration by __float on Sunday June 30, 2024 @05:20PM (#40840482)
They said it was stored with Git LFS, so it's just a pointer in the repo, not all 80 MB.
Re:The Great Database Migration by candiddevmike on Sunday June 30, 2024 @05:27PM (#40840541)
> That's... clever

It's tech debt

Re:The Great Database Migration by mbreese on Sunday June 30, 2024 @07:40PM (#40841359)
It’s tech debt, but it’s cheap debt. 80Mb isn’t much to have to worry about. There are many ways to use a DB of that size that will be “good enough”. So long as the queries were returning fast enough and the dev overhead wasn’t too much (LFS was a good choice), I’m not sure I’d have worried much.

There is tech debt that will eat away at you, and then there is this… something that you’ll want to replace someday, but it can wait until you are ready to tackle it properly. And you may not even know what data access patterns you need right away. For a new company with a pretty new way of working with insurance, this seems like a good trade off to me. Plus, it seems clear that they weren’t confident in their backend engineering yet, so this gave them time to figure things out.

Re:The Great Database Migration by gfody on Sunday June 30, 2024 @08:02PM (#40841458)
80mb is nothing, I think it's a no brainer to keep it in git but they should've commited the .dump instead of the .db
Re:The Great Database Migration by agent001 on Sunday June 30, 2024 @10:12PM (#40842013)
Neon actually solved a bunch of problems here. Sure, 80MB might not seem huge, but in a git repo? Also, 80MB is actually large enough to impact git performance according to Github(https://docs.github.com/en/repositories/working-with-files/m...). That can be a pain, especially if you're updating it often. Neon let them ditch the whole 'DB-in-git' approach. No more slowing down the repo or worrying about it ballooning with every update. Plus, it probably made deployment and scaling way smoother. Sometimes the 'clever' solution isn't the best long-term, you know? Kudos to them for recognizing that and making the switch.