The biggest difference in these benchmarks comes down to how multiple threads are leveraged; (disclaimer: primary CSV.jl author here).

In CSV.jl, it was relatively straightforward to add multithreaded parsing support; we chunk up the file, find row starts, then spawn a threaded task to process each chunk. In data.table (fread), they have a constraint of R having a global String intern store; so it fundamentally restricts the multithreading capabilities by requiring strings to be interned sequentially. In pandas, there are similar nontrivialities jumping between the C++ source and python object world that adds a lot of complexity (and probably explains why no one has made the effort to do multithreaded parsing). It's interesting to note, however, that the apache arrow project (pyarrow package in Python), has a multithreaded csv parser integrated with the project. It provides similar performance to CSV.jl because it was built from the ground up to process chunks on multiple threads into "arrow tables". Unfortunately, it's tied very specifically to the arrow project, so pandas doesn't benefit from its work! It's one of my favorite features about Julia that CSV.jl automatically integrates with other "data table formats"; i.e. ODBC.jl, SQLite.jl, MySQL.jl, Arrow.jl, DataFrames.jl, etc. They can all leverage CSV.jl as "their csv parser" because it's seamlessly integrated via well-established "table" apis.

Because it is not a bottleneck. Modern SSDs can read data at more than 500 megabytes per second, which, on a single thread gives you just about 6-8 clock cycles to process a single byte. That’s not a lot of CPU instructions so it will take quite a number of threads to handle this bandwidth.

The SSD in my current workstation tops out at 5GB/s sequential reads and 680kIOPS. That's a modestly high-end consumer M.2 drive, not enterprise exotica. SSDs are really, really fast.

Yeah Intel optane and the like will be even faster.

Without having looked at it, it might also be a case that the other libraries spend more time handling more obscure corner cases and doing more robust error handling.