News for hackers. Stuff that matters.
Re:Friends don't let friends export to CSV by askvictor on Tuesday March 26, 2024 @03:34AM (#39825083)
We have a use case where we effectively need to have a relational database, but in git. The database doesn't change much, but when it does, references between tables may need to be updated. But we need to easily be able to see diffs between different versions. We're trying an SQLite DB, with exports to CSV as part of CI - the CSV files are human-readable and diff'able.

It's also worth noting that SQLite can ingest CSV files into memory and perform queries on them directly - if the files are not too large, it's possible to bypass the sqlite format entirely.

Re:Friends don't let friends export to CSV by john-shaffer on Tuesday March 26, 2024 @09:38AM (#39827665)
Have you considered https://github.com/dolthub/dolt for your use case?
Re:Friends don't let friends export to CSV by thenbe on Tuesday March 26, 2024 @09:39AM (#39827683)
> we need to easily be able to see diffs between different versions

Can git attributes help in this case? It allows you to teach git how to diff binary files using external tools. Here [0] is a demonstration for teaching git to produce an "image diff" for *.png files using exiftool. You can do something similar for *.sqlite files by adding these 3 lines [1] [2]. The sqlite3 cli needs to be installed.

Alternatively, there's a tool that might also fit the bill called datafold/data-diff [3]. I'm pretty sure I originally heard of it on a HN thread so those comments may offer even more alternative solutions.

[0]: https://youtu.be/Md44rcw13k4?t=540 [the relevant timestamp is @ 9:00]

[1]: https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8...

[2]: https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8...

[3]: https://github.com/datafold/data-diff

Re:Friends don't let friends export to CSV by zachmu on Wednesday March 27, 2024 @07:21PM (#39845823)
Somebody already said this, but we built exactly this and it's called Dolt.

https://github.com/dolthub/dolt

Would love to hear how it addresses your use case or falls short.