News for hackers. Stuff that matters.
Re:Asynchronous I/O in DuckDB: Work, Thread, Work by pdet on Sunday August 16, 2026 @06:23AM (#49318687)
DuckDB uses a speculative parallel CSV parsing technique. The basic idea is that the parser speculates about the state the CSV parser is in at a random byte (e.g., whether it is inside a quoted field) and tries to figure out where the next row starts based on that.There are validation steps during finalization as well, to ensure the parser did not got anything wrong in its speculation.

I've never gotten around to writing a blog post about it, but I go quite in-depth on the technique in this presentation: https://www.youtube.com/watch?v=YrqSp8m7fmk

(Disclaimer: I'm the author of the blog post and also the developer who implemented the entire CSV parser in DuckDB.)

Re:Asynchronous I/O in DuckDB: Work, Thread, Work by hilariously on Sunday August 16, 2026 @07:12AM (#49318941)
Burning my points to say dude that's sick, parsing CSVs is hell I am genuinely going to watch this thank you.
Re:Asynchronous I/O in DuckDB: Work, Thread, Work by _zoltan_ on Sunday August 16, 2026 @10:43AM (#49320601)
there isn't any upvote limit on HN.
Re:Asynchronous I/O in DuckDB: Work, Thread, Work by hilariously on Sunday August 16, 2026 @08:00PM (#49325075)
No, but its genuinely looked down upon to make frivolous comments.
Re:Asynchronous I/O in DuckDB: Work, Thread, Work by abirch on Sunday August 16, 2026 @06:32AM (#49318740)
Thank you so much. I love DuckDB!
Re:Asynchronous I/O in DuckDB: Work, Thread, Work by pletnes on Sunday August 16, 2026 @09:25AM (#49319844)
You’re officially my hero, the duckdb csv parser is the fastest I could find and helped save a data project with hundreds of gigabytes of csv pain.
Re:Asynchronous I/O in DuckDB: Work, Thread, Work by quag on Sunday August 16, 2026 @06:04PM (#49324178)
Thank you. I love using duckdb and frequently query directly from zstd compressed files (and write out new ones). They’re easy to inspect using visidata and make outer use of. I found the talk interesting and enjoyable.