News for hackers. Stuff that matters.

A comment on Go can only read 1GiB per Read call. Parent comment

Re:Go can only read 1GiB per Read call by rini17 on Saturday February 10, 2024 @02:55AM (#39324434)
Or not, because it simplifies code, no differences between platforms. And if syscall overhead once per each GB is really a problem you should probably try mmap or some other IO style.
[ Parent | on HN ]
Re:Go can only read 1GiB per Read call by hmottestad on Saturday February 10, 2024 @04:01AM (#39324702)
Mmap seems specific to POSIX according to this post[1], so it's not a portable way of using memory mapped files. Since there are already differences between platform capabilities in golang there really shouldn't need to be an artificial limit to how large of a chunk you can read in one operation. In 20 years I'm sure 1 GiB will be considered small.

1: https://stackoverflow.com/questions/52589949/undefined-sysca...

[ Parent | on HN ]
Re:Go can only read 1GiB per Read call by oflebbe on Saturday February 10, 2024 @03:03AM (#39324464)
Only that there is no official mmap support in go, due to gc concerns, afaik
[ Parent | on HN ]
Re:Go can only read 1GiB per Read call by barsonme on Saturday February 10, 2024 @03:06AM (#39324480)
Hm? https://pkg.go.dev/golang.org/x/sys/unix#Mmap
[ Parent | on HN ]
Re:Go can only read 1GiB per Read call by hmottestad on Saturday February 10, 2024 @03:35AM (#39324583)
The author saw performance issues with this restriction, which is why they wrote the post.
[ Parent | on HN ]
Re:Go can only read 1GiB per Read call by xyzzy123 on Saturday February 10, 2024 @03:36AM (#39324587)
I would be curious to see the numbers on that.
[ Parent | on HN ]
Re:Go can only read 1GiB per Read call by hmottestad on Saturday February 10, 2024 @03:41AM (#39324613)
You should probably ask the author then. There are a bunch of links to various socials on the bottom of the post.
[ Parent | on HN ]
Re:Go can only read 1GiB per Read call by technicolor on Saturday February 10, 2024 @07:27AM (#39325676)
Hello, author here. Just to clarify, I wasn't saying that this has any perf impact per se. I was trying to see why syscall showed up quite a bit in the profile, when reading a large file, no matter what buffer size I pass in. Rather than spend time finishing the actual work, I spent way too much time trying to understand the reason for the limit.
[ Parent | on HN ]