TomHubelbauer
TomHubelbauer on Hacker News
Why don't you hang around for a week and see for yourself?
They've got a pretty large audience, as it appears, large enough to bring enough attention to this, so while it definitely was a risk they must've been weighing heavily to publish this (being so different from their usual stuff), I'm glad they did.
Very cool insight, do you know if the Windows kernel does the same? I'd like to learn more about it in context of Windows first, then Linux.
I didn't know that, thanks for sharing! It's really interesting and might come in handy to know this.
Aside from the mobile technologies you mention, I also wish a filter for .NET was present. Granted, maybe no companies in the original dataset do .NET, but if I could filter for .NET and see no results, I would know for sure, currently I just can't tell.
EDIT: On second thought you basically answered my question in the last paragraph of your reply, I'm keeping it up in case you want to confirm my understanding, but the way I read it, the 15 % speed up comes from the fact that indeed CPU processing (the regexes) was comparable to the file reading in speed and breaking up the bits to individual core caches is what gives way for the speed up, memory is fast enough for this to be possible.
I don't understand the I/O part of the whole deal. AFAIK you can't read multiple files at once off the drive, so are the files read sequentially to memory and then searched in parallel in memory? Does memory allow you to read multiple places at once, even with multiple threads? Surely the files are too large for the processor caches to be of any effect. So if you'd entertain my curiosity a bit more, is what's happening the fact that multiple files are cached in memory and memory is so fast that loading bits of the files to the processor cache takes less time than regexing those bits, moving the balance of this process to the CPU-bound side? Is the actual parallelism in the fact that multiple cores can search their individual caches at the same time and loading those caches from the RAM is fast enough to not become a bottleneck? Sorry for possibly amateurish question, I've never dug deep enough when it comes to parallelism to understand this, but I spent a good amount of time thinking about parallelising I/O stuff and came to the conclusion that I/O must be magnitudes slower and thus always is a bottleneck and any sort of I/O-bound problem (which file search surely is) must be non-parallelizable and instead can only be sped up by keeping indexes the way some OS's do.
Can someone explain how searching files is "embarrassingly parallel"? I imagine storage drive to be sequential, the reading head can only be in one place at the time. For SSDs, this may be different, that I don't know. Does this speed up only apply to SSDs? Or is the I/O not the bottleneck here and instead the project benefits from the searching of memory cached files being somehow well parallelizable (memory still seems basically sequential, but is much faster, maybe fast enough to make this problem overwhelmingly CPU-bound?)