Re:The Linux kernel's inability to gracefully handle low mem...
by caf on Tuesday August 06, 2019 @02:25AM
(#20622690)
If you are reading unbuffered (ie. O_DIRECT) then you are reading directly into the memory block the user supplied, so you cannot read ahead - there's nowhere to put the extra data.
[ Parent
| on HN ]
Re:The Linux kernel's inability to gracefully handle low mem...
by paulsutter on Tuesday August 06, 2019 @07:32AM
(#20624201)
Of course you can read ahead using multiple buffers, you can issue as many reads as you want concurrently
Re:The Linux kernel's inability to gracefully handle low mem...
by caf on Tuesday August 06, 2019 @10:28AM
(#20625450)
I think it is pretty clear I was referring to kernel-mediated readahead. Sure, you can achieve the same thing in userspace using async IO or threads.
Re:The Linux kernel's inability to gracefully handle low mem...
by paulsutter on Tuesday August 06, 2019 @11:25AM
(#20625960)
I think it’s pretty clear that I was referring to reading large files using direct io, and your comment was totally irrelevant
Re:The Linux kernel's inability to gracefully handle low mem...
by caf on Tuesday August 06, 2019 @08:21PM
(#20630953)
Reading large files using direct IO defeats kernel read-ahead, which means you have to take on the complexity of reimplementing it in userspace, or the performance hit of not having it.
This is a good reason for programs to use buffered IO even when they are reading a large file once, so yes my comment was entirely on point.
Re:The Linux kernel's inability to gracefully handle low mem...
by paulsutter on Wednesday August 07, 2019 @11:52AM
(#20635830)
Your comments were misleading, even if you want to argue they were on point. I know that you're knowledgable about the kernel, I ready your other comments. The following comments are just not true, except in bizarro world where all code must be in the kernel:
- "Readahead... needs to be buffered"
- "reading unbuffered (ie. O_DIRECT)... you cannot read ahead"
Good application code needs to work around kernel limitations all the time. Your suggestion of fadvise is reasonable, except in practice fadvise rarely makes a measurable difference. The only way to get what you really want is to code it the way you want it, and fortunately, there are multiple ways to do that.