Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ...
by thangngoc89 on Monday May 20, 2024 @07:42AM
(#40414408)
That's why you always test your backup. I backup the full sqlite.db every day and test the litestream replication every week. So far litestream have been solid.
[ Parent
| on HN ]
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ...
by mickeyp on Monday May 20, 2024 @07:56AM
(#40414483)
Would you care to tell us what your backup and restore policy would be for 1.7 PB of data?
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ...
by thangngoc89 on Monday May 20, 2024 @08:03AM
(#40414527)
I'm replying to the question of how one would replicate SQLite 3 in production for redundancy. I myself consider 10GB would be the limit for using SQLite 3 in read/write in production and switch to PostgreSQL.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ...
by sgt on Monday May 20, 2024 @09:14AM
(#40415110)
That's a huge discrepancy. One half of HN wants to put petabytes on SQLite, while your limit is only 10GB.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ...
by Closi on Monday May 20, 2024 @10:21AM
(#40415753)
Why not use SQLite's own guidance on where SQLite probably isn't appropriate:
- Client/Server applications (Check)
- High-volumes (Check)
- Large datasets (Check)
- High concurrency, particularly for writes (Check)
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ...
by squeaky-clean on Tuesday May 21, 2024 @03:12AM
(#40425109)
There's only one person in this thread trying to put petabytes in SQLite. Everyone else is telling them the myriad reasons why it's a terrible idea.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ...
by zaphirplane on Monday May 20, 2024 @07:50AM
(#40414444)
By the time the TB is restored, time to start the next test
How do you detect restored but bit flipped data ?
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ...
by thangngoc89 on Monday May 20, 2024 @08:00AM
(#40414509)
I do this in backup testing:
sqlite3 /path/to/db
sqlite> PRAGMA integrity_check;
See SQLite3 documentation: https://www.sqlite.org/pragma.html#pragma_integrity_check
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ...
by zaphirplane on Monday May 20, 2024 @08:21AM
(#40414672)
Sounds like it will take awhile for TB and it checks db integrity not data integrity