Re:The One Billion Row Challenge in Go: from 1m45s to 4s in ...
by geysersam on Sunday March 03, 2024 @04:08AM
(#39579559)
Sounds very reasonable. In the blog post about 20s were shaved off by assuming we don't need complicated string parsing. An of the shelf library can't make that assumption so they will always have to pay the extra cost.
Re:The One Billion Row Challenge in Go: from 1m45s to 4s in ...
by 392 on Sunday March 03, 2024 @06:23AM
(#39580166)
True in general but some (especially libs aimed at larger datasets processed in batch) are taking advantage of benchmarks like this to do things like:
Try the fast way, if it works great
Try the slow way, if above fails
This makes the slow path 2x slower at worst (and you can advise to always use the slow way with optional params) but the fast path can be 10x faster
Re:The One Billion Row Challenge in Go: from 1m45s to 4s in ...
by sharno on Sunday March 03, 2024 @02:09AM
(#39579093)
That’s the question worth asking imo. I was wondering how fast is the idiomatic Java solution
Re:The One Billion Row Challenge in Go: from 1m45s to 4s in ...
by kaba0 on Sunday March 03, 2024 @03:58AM
(#39579520)
Is that the same hardware? Otherwise it doesn’t say much.
Re:The One Billion Row Challenge in Go: from 1m45s to 4s in ...
by KingOfCoders on Sunday March 03, 2024 @04:40AM
(#39579697)
Java is often slightly faster than Go, has similar (perhaps, older, better optimized Map) constructs, perhaps better GC (older, more optimized), though I don't think the GC is a challenge, has slower startup times - so I'd say roughly the same as the idiomatic Go version?
Re:The One Billion Row Challenge in Go: from 1m45s to 4s in ...
by kaba0 on Sunday March 03, 2024 @05:45AM
(#39579999)
Java’s GC is just incomparably better, literally every research on the topic is written for Java. Though that’s true that without value types, Java does rely slightly more on it.
Re:The One Billion Row Challenge in Go: from 1m45s to 4s in ...
by geodel on Sunday March 03, 2024 @02:21PM
(#39583543)
It could be matter of fact comment if incomparably better and slight reliance are brought to the middle.
Re:The One Billion Row Challenge in Go: from 1m45s to 4s in ...
by Xeoncross on Sunday March 03, 2024 @01:37PM
(#39583102)
Java can be faster than Go, but it comes at the same cost as most "faster" things in software: Java uses significantly more memory.
Java is also more mature, which means you are entering a massive package-bloat setup that has evolved over the years to work for everyones wild and varied needs. By the time you have your database, cache, http/other handlers, tests, fixtures, metrics, logging, tracing, etc... setup you're looking at a scary pile of dependencies spanning thousands of classes that would make even NPM jealous.
Re:The One Billion Row Challenge in Go: from 1m45s to 4s in ...
by benhoyt on Sunday March 03, 2024 @05:21AM
(#39579897)
On my machine the simple/idiomatic "baseline" Java solution (https://github.com/gunnarmorling/1brc/blob/main/src/main/jav...) takes 2m0s, compared to 1m45s for my simple Go version.
Re:The One Billion Row Challenge in Go: from 1m45s to 4s in ...
by gigatexal on Sunday March 03, 2024 @02:37AM
(#39579212)
Where’s the source data I’d like to attempt ingesting this and processing it with DuckDb.
Re:The One Billion Row Challenge in Go: from 1m45s to 4s in ...
by schu on Sunday March 03, 2024 @03:14AM
(#39579346)
Instructions on how to create it can be found here:
https://github.com/gunnarmorling/1brc?tab=readme-ov-file#run...
The Python version:
https://github.com/gunnarmorling/1brc/blob/main/src/main/pyt...
Re:The One Billion Row Challenge in Go: from 1m45s to 4s in ...
by nhinck3 on Sunday March 03, 2024 @03:11AM
(#39579329)
In the original 1BRC, it's a python script that generates the data.
Re:The One Billion Row Challenge in Go: from 1m45s to 4s in ...
by llimllib on Sunday March 03, 2024 @11:50AM
(#39582010)
Here’s a thread on results with duckdb, I don’t mean to discourage you taking a shot at all though: https://github.com/gunnarmorling/1brc/discussions/39
Re:The One Billion Row Challenge in Go: from 1m45s to 4s in ...
by jsmith99 on Sunday March 03, 2024 @05:00AM
(#39579796)
I’m surprised at the poor performance of python here. For reference there are several very brief R examples which are just 2-3 seconds. Eg http://blog.schochastics.net/posts/2024-01-08_one-billion-ro...
Re:The One Billion Row Challenge in Go: from 1m45s to 4s in ...
by cmdlineluser on Monday March 04, 2024 @10:32AM
(#39591618)
Are you talking about the 2nd table in the Benchmark section?
It seems they are not running against the full dataset:
> Moving on to the 100 million file to see if size makes a difference.
ggplot2::autoplot(reorderMicrobenchmarkResults(bench1e8))
One would also have to run both approaches on the same hardware for a meaningful comparison?
I’m kind of interested in the opposite problem, what is the simplest solution using a well known library/db that approaches the fastest hand optimized solution to this problem?