Dat – Distributed Dataset Synchronization and Versioning

Re:Dat – Distributed Dataset Synchronization and Versioning by justinclift on Friday May 26, 2017 @09:55AM (#14424036)

Hmmm, is the data set something which would fairly naturally fit in a series of SQLite databases?

100GB is way too large for the project I'm working on at the moment (dbhub.io), as even a bunch of people downloading something that large would nuke our sponsorship budget since we're just starting out (still pre-launch).

However, if we gain traction and become cash positive, data sets this size would be good to cater to. :)

[ Parent | on HN ]

Re:Dat – Distributed Dataset Synchronization and Versioning by rspeer on Friday May 26, 2017 @10:39AM (#14424351)

I used to keep it in SQLite (much easier to distribute than PostgreSQL). It worked a lot better than many other options I tried. However, rebuilding the database from updated data would take more than a day, and some queries were too slow.

Switching to PostgreSQL sped things up, at the cost of requiring a separate database process, dealing with psql's weird access control, and adding an inconvenient step of loading the data using COPY commands.

[ Parent | on HN ]

Re:Dat – Distributed Dataset Synchronization and Versioning by justinclift on Friday May 26, 2017 @01:23PM (#14425720)

Hmmm, sounds like the data itself would be feasible then. SQLite could be considered just a data transport container for this purpose. :)

[ Parent | on HN ]