The Billion Row Challenge (1BRC) – Step-by-Step from 71s to 1.7s

Re:The Billion Row Challenge (1BRC) – Step-by-Step from 71s ... by Keyframe on Thursday February 22, 2024 @04:12PM (#39473321)

I must admit I only glanced at the article and in the past when topic appeared. How does this work, the speed of 1.7s I mean? Taking a look at input, let's say average entry is 16 bytes, there's billion of it; That's ~16GB. Average read speed of SSDs is what, ~500MB/s? Scanning alone at max throughput will take half a minute. This must be relying on DDR4+ read speeds which would probably come in under a second in certain cases. Is that's what's going on, RAM disk?

[ Parent | on HN ]

Re:The Billion Row Challenge (1BRC) – Step-by-Step from 71s ... by haxen on Thursday February 22, 2024 @04:19PM (#39473403)

I used to benchmark a lot on an enterprise-grade SSD 10 years ago, and that was already at 2 GB/s. Today, even my laptop's SSD supports multiple GB/s.

But you're right about the contest -- each program was measured five times in a row, and there was enough RAM to fit the entire file into the page cache.

The best time using all 32 cores (64 hyperthreads) on the evaluation machine was 323 milliseconds.

[ Parent | on HN ]

Re:The Billion Row Challenge (1BRC) – Step-by-Step from 71s ... by Keyframe on Thursday February 22, 2024 @04:30PM (#39473543)

I went to the original repo, it's indeed a RAM disk. That makes it clear then. I was already almost excited about some IO wizardry going on.

Results are determined by running the program on a Hetzner AX161 dedicated server (32 core AMD EPYC™ 7502P (Zen2), 128 GB RAM).

Programs are run from a RAM disk (i.o. the IO overhead for loading the file from disk is not relevant), using 8 cores of the machine.

[ Parent | on HN ]

Re:The Billion Row Challenge (1BRC) – Step-by-Step from 71s ... by haxen on Thursday February 22, 2024 @04:38PM (#39473635)

Running without RAM cache would be a great followup to this challenge. I think a time around 2-3 seconds should be achievable. But, it would be highly sensitive to the hardware setup and how well the disk-reading code is placed on cores relative to the connection to the disk. Not sure what it would take to allow hundreds of contestants to benefit from the best arrangement.

[ Parent | on HN ]

Re:The Billion Row Challenge (1BRC) – Step-by-Step from 71s ... by pulse7 on Thursday February 22, 2024 @04:32PM (#39473566)

PCIe 3.0 x4 SSD runs at ~3500MB/s, PCIe 4.0 x4 SSD runs at ~7000MB/s and PCIe 5.0 x4 SSD runs at ~14000MB/s (yes - megabytes, not megabits)!

[ Parent | on HN ]

Re:The Billion Row Challenge (1BRC) – Step-by-Step from 71s ... by sroussey on Thursday February 22, 2024 @05:51PM (#39474448)

1 KIOXIA SSD on PCI 5 can do that 14000MB/s and 2M IOPS on 4 lanes. Some servers have 100+ lanes. You can have some serious speed if you want!

[ Parent | on HN ]

Re:The Billion Row Challenge (1BRC) – Step-by-Step from 71s ... by Keyframe on Thursday February 22, 2024 @04:44PM (#39473715)

blazing! I had ye olde SSD SATA bricks in mind though.

[ Parent | on HN ]

Re:The Billion Row Challenge (1BRC) – Step-by-Step from 71s ... by CyberDildonics on Thursday February 22, 2024 @08:54PM (#39475934)

14000MB/s? That's around 14,000,000 KB/s !

[ Parent | on HN ]