rkaveland

Author here. I will be trying out Kagi for some time. Search is worth paying money for. Thanks for the heads-up!

Author of the blog post here -- that looks like exactly what I needed, so I'm probably going to add a dependency to it to https://github.com/kaaveland/eugene/ so I can delete a ton of code. :-)

That looks fantastic, so I'm actually just going to put a link to it in the post so that more people see it.

A while back, I wrote a blog post that experimented with doing transactional DDL in postgres, and investigating the state of `pg_locks` to check if statements were likely to interfere with queries. I've expanded on this idea and made a tool, that I named `eugene` for the title of the first blog post.

Check out the examples in https://github.com/kaaveland/eugene/tree/main or read the blog post -- I would love to have more ideas for where to take this, both concrete migration hints that `eugene` could show people, but also functionality and ways to integrate it into CI.

This is super cool, I love how it suggests safer ways to achieve the same goal for many common risky migrations. Definitely going to sit down and study these.

Thanks, I will fix this part. As far as I can tell, it is only the example of what can cause the table rewrite that's wrong, so it would be correct if the example was to make a varchar shorter? Although I guess that's not a table rewrite, only a long validation...

> Well what would be a more accurate title? "CSV format should only be for external interchange or archival; columnar formats like Parquet or Arrow better for performance"?

Something more boring, like "Consider whether other options make more sense for your data exports than CSV". Plenty of people have suggested other good options in comments on this submission, such as for example sqlite. I think the post comes off as if I'm trying to sell a particular file format for all uses cases, when what I had in mind when writing it was to discourage using CSV as a default. CSV has a place, certainly, but it offloads a lot of complexity on the people who are going to consume the data, in particular, they need to figure out how to interpret it. This can't necessarily be done by opening the file in an editor and looking at it, beyond a certain size you're going to need programming or great tools to inspect it anyway.

I was given an initial export of ~100 poor quality CSV files totaling around 3TB (~5-6 different tables, ~50 columns in each) in size a few years back, and had to automate ingestion of those and future exports. We could've saved a lot of work if the source was able to export data in a friendlier format. It happened more than once during that project that we were sent CSVs or Excel sheets that had mangled data, such as zip codes or phone numbers with leading 0s removed. I think it is a good thing to inform people of these problems and encourage the use of formats that don't necessity guessing data types. :shrug:

> People are busy; instead of hinting "something more robust than CSV", mention the alternatives and show a comparison (load time/search time/compression ratio) summary graph. (Where is the knee of the curve?)

This might be an interesting thing to follow up later, but would require a lot more work.

I regret forgetting about it in the article. sqlite is a great solution.

sqlite is an excellent choice.

Author here. I see now that the title is too controversial, I should have toned that down. As I mention in the conclusion, if you're giving parquet files to your user and all they want to know is how to turn it into Excel/CSV, you should just give them Excel/CSV. It is, after all, what end users often want. I'm going to edit the intro to make the same point there.

If you're exporting files for machine consumption, please consider using something more robust than CSV.