It can make huge amounts of difference in a production system; where I work, we process terabytes of csv data every day; saving minutes per file can add up to enormous differences in CPU cost/time for a production system running 24/7.

I agree that for a data scientist doing exploratory analysis locally on their computer, it doesn't make nearly as much a difference (also because they're usually not working on crazy large files).

The performance work in the CSV.jl package (that the article is about) was very much geared towards these kinds of production scenarios.

Right - sounds like you have more of a production support role vs. a data analysis workflow kind of task. Tacking on "exploratory" is helpful but I'm still concerned that you misuse the overall concept of analysis. It's decision-making task, which is practically the opposite of production support.

>Tacking on "exploratory" is helpful but I'm still concerned that you misuse the overall concept of analysis. It's decision-making task, which is practically the opposite of production support.

Why should the exploratory and production teams be using completely different tools? That seems like it would cause frictions in productivity and make there be gaps that introduce translation errors. I would venture to say that just having the exploratory and production teams working using the same code base is a very strong productivity gain, and we've seen this is true in many companies.

One is R&D, and one is operations. Obviously there are situations where it makes sense to combine them, but most successful companies with any real technology to sell will not treat R&D this way.

> process terabytes of csv data every day

All stored on NVMe SSDs? Because unless you have really fast IO the CSV parser isn’t going to be the bottleneck.

I am curious: how do you transfer these amounts of data fast?

Unfortunately almost exclusively via http rest apis. It's not great, but it's the lowest common denominator between the vast "ingestion" service we've built (connectors to web apis, local application for local file upload, raw api endpoints, etc.).

We've started exploring the apache arrow format as a compressible binary format with a dedicated wire format just to cut down on parsing processing costs.

I can think of many cases where that is useful. It is important to remember not all workflows and needs are the same as yours.

I don’t really understand tour web dev comment either. Web devs are notoriously not interested in performance.

Maybe I don't understand web dev; only meant that when you have many users besides yourself (analyzing data) then performance matters more.

It's important to remember, wtf?

Then it goes both ways right? I think we need to also say "For most people Python is fine, if you need that extra speed, try Julia" instead of blanket praises of Julia as true competitor to Python (It doesn't even come close IMO).