News for hackers. Stuff that matters.
Re:SQLite Is Serverless by hinkley on Sunday January 26, 2020 @02:35PM (#22154188)
Do you generate the file from scratch every time or do you modify the previous one as new data arrives?
Re:SQLite Is Serverless by tracker1 on Sunday January 26, 2020 @02:45PM (#22154249)
Depends on what you want... if you have a separate db project, you can have the output of that project be a clean database for testing other things, or a set of migration scripts for existing deployments.

I've been working on doing similar with containerized dababase servers for testing, while still having versioned scripts for prod (multiple separate deployments).

Re:SQLite Is Serverless by at_a_remove on Sunday January 26, 2020 @03:46PM (#22154672)
It is a bit of a hybrid.

In the early stages of development of whatever the ETL process is, I keep the database and just empty it out each time. As I got more of a sense of what I needed, I started DROPing my TABLEs more often and remaking them. Eventually I would make the whole database from scratch once I was along the way and had most everything fleshed out.

Re:SQLite Is Serverless by hinkley on Sunday January 26, 2020 @06:00PM (#22155459)
Ok. So each export is a full dump, not a delta on a previous one.

Do you anticipate hitting a wall at some point where the total time becomes a problem?

Re:SQLite Is Serverless by at_a_remove on Sunday January 26, 2020 @06:08PM (#22155504)
Well, it depends on the process. Some were full dumps, some were deltas pushed up to the final database, sometimes both (this product in particular had a load from file capability that you were supposed to use but some edge cases that were not well-addressed).

No, the time never grew significantly.

For one of the analysis projects, just one step of the analysis was quite time consuming but it would have been that way no matter what. SQLite allowed me to let it grind away overnight (or even over a weekend) on a workstation without tormenting production servers.