Ok. So each export is a full dump, not a delta on a previous one.
Do you anticipate hitting a wall at some point where the total time becomes a problem?
Re:SQLite Is Serverless
by at_a_remove on Sunday January 26, 2020 @06:08PM
(#22155504)
Well, it depends on the process. Some were full dumps, some were deltas pushed up to the final database, sometimes both (this product in particular had a load from file capability that you were supposed to use but some edge cases that were not well-addressed).
No, the time never grew significantly.
For one of the analysis projects, just one step of the analysis was quite time consuming but it would have been that way no matter what. SQLite allowed me to let it grind away overnight (or even over a weekend) on a workstation without tormenting production servers.
In the early stages of development of whatever the ETL process is, I keep the database and just empty it out each time. As I got more of a sense of what I needed, I started DROPing my TABLEs more often and remaking them. Eventually I would make the whole database from scratch once I was along the way and had most everything fleshed out.