News for hackers. Stuff that matters.
Re:I/O is no longer the bottleneck by imtringued on Saturday November 26, 2022 @08:15AM (#33752357)
The file is only being read once according to the blog and then it is in memory. This isn't an I/O intensive application at all.

I have written a small script in python that does something similar. I have a word list with 1000 words and I check the presence of the words. Here is the thing. For every word I go through the entire file. So lets say I scan the file one thousand times. In fact I did something more complicated and ended up going 6000 times over the original file and yet it still took only three seconds. If all these scans had to reread the file it would take forever.

Re:I/O is no longer the bottleneck by kragen on Saturday November 26, 2022 @09:30AM (#33752919)
he has figures for both cached and uncached performance

if all your scans had to reread the file from nvme it would take five times as long if we extrapolate from those figures

not forever

Re:I/O is no longer the bottleneck by Severian on Saturday November 26, 2022 @10:16AM (#33753312)
This sounds like a very inefficient method. The Python stdlib module 'collections' includes a Counter class which can do this for you. Then all you need to do is copy the dict key/item pair to a new one that match your word list.
Re:I/O is no longer the bottleneck by nunez on Saturday November 26, 2022 @12:14PM (#33754522)
the optimized version does it byte by byte