I have had many other situations where CSV was the bottleneck.
I still would default to CSV first in many situations because it's robust and easily inspected by hand.
How much RAM do you have? What's the ratio of [smallest CSV file which bottlenecks]/[your RAM]?
That being said, the real point here is that folks blindly use CSVs for internal-facing processes even though there's no particular reason to, and they have plenty of drawbacks. If you're just building some kind of ETL pipeline why wouldn't you use Parquet? It isn't as if you're opening stuff in Excel.
If the title was “friends in certain circumstances shouldn’t let friends in certain circumstances export to csv.”
Even a laptop with 8gb ram can open a gig csv.
Of course the internals of your etl will use some efficient data structure, but you’d still want to export as csv at some point to get data to other people. Or you want your friends to export csv to get data to you.
Csv is not very performant, but it doesn’t matter for these use cases.
I’ll also add that I’m not working with the csvs, they are just I/o. So any memory issues are handled by the load process. I certainly don’t use csvs for my internal processes. Just for when someone sends me data or I have to send it back to them.
That being said my workstation is pretty big and can handle 10s of gigs of csv before I care. But that’s usually just for dev or debugging and anything that sticks around will be working with data in some proper store (usually parquet distributed across nodes).