The Billion Row Challenge (1BRC) – Step-by-Step from 71s to 1.7s

Re:The Billion Row Challenge (1BRC) – Step-by-Step from 71s ... by haxen on Thursday February 22, 2024 @04:19PM (#39473403)

I used to benchmark a lot on an enterprise-grade SSD 10 years ago, and that was already at 2 GB/s. Today, even my laptop's SSD supports multiple GB/s.

But you're right about the contest -- each program was measured five times in a row, and there was enough RAM to fit the entire file into the page cache.

The best time using all 32 cores (64 hyperthreads) on the evaluation machine was 323 milliseconds.

[ Parent | on HN ]

Re:The Billion Row Challenge (1BRC) – Step-by-Step from 71s ... by Keyframe on Thursday February 22, 2024 @04:30PM (#39473543)

I went to the original repo, it's indeed a RAM disk. That makes it clear then. I was already almost excited about some IO wizardry going on.

Results are determined by running the program on a Hetzner AX161 dedicated server (32 core AMD EPYC™ 7502P (Zen2), 128 GB RAM).

Programs are run from a RAM disk (i.o. the IO overhead for loading the file from disk is not relevant), using 8 cores of the machine.

[ Parent | on HN ]

Re:The Billion Row Challenge (1BRC) – Step-by-Step from 71s ... by haxen on Thursday February 22, 2024 @04:38PM (#39473635)

Running without RAM cache would be a great followup to this challenge. I think a time around 2-3 seconds should be achievable. But, it would be highly sensitive to the hardware setup and how well the disk-reading code is placed on cores relative to the connection to the disk. Not sure what it would take to allow hundreds of contestants to benefit from the best arrangement.

[ Parent | on HN ]