News for hackers. Stuff that matters.
Re:Searching 20 GB/sec: Systems Engineering Before Algorithms by to3m on Thursday May 08, 2014 @07:46PM (#7718448)
Random rant time!

You have to code for Windows just so. Dear POSIX people in general, there is more to cross platform coding than being able to build it with cygwin.

Every grep I've used on Windows handles wildcards, so it must be doing the expansion itself. The right way to do this sort of thing on Windows is to call FindFirstFile/FindNextFile, which is a cross between readdir and stat in that each call returns not only the name of one directory entry that matches the pattern (as per readdir+fnmatch) but also its metadata (as per stat). So, if you were going to call readdir (to get the names) and then stat each name (to get the metadata), you should really be doing all of it in just this one loop so that each bit of data is retrieved just the once.

But this is exactly what POSIX programmers would seemingly never, ever do. Probably because doing it that way would involve calling a Win32 function. Well, more fool them, because getting this data from FindFirstFile/FindNextFile seems pretty quick, and getting it any other way seems comparatively slow.

I cobbled together a native Win32 port of the_silver_searcher a while ago and it was about twice as fast as the stock EXE thanks to retrieving all the file data in one pass. In this case the readdir wrapper just needed to fill in the file type in the dirent struct; luckily it seems that some POSIX-style systems do this, so there was already code to handle this and it just needed the right #define adding. (I have absolutely no idea why MingW32 doesn't do this already.)

Prior to switching to the_silver_searcher, I used to use grep; the grep I usually used is the GNU-Win32 one (http://gnuwin32.sourceforge.net/packages/grep.htm), and it looks to call readdir to get the names, and then stat to get each name's details. I checked that just now, and after all those years I can finally imagine why it's so slow.

GNU-Win32 find is really slow, too.

Re:Searching 20 GB/sec: Systems Engineering Before Algorithms by herf on Friday May 09, 2014 @10:52AM (#7721160)
Yes don't call GetFileAttributes (which your POSIX layer does to emulate grep) - especially over a network (samba) connection.

But the original point is OPEN is expensive, which it really is, especially over a network connection.

Re:Searching 20 GB/sec: Systems Engineering Before Algorithms by lucian1900 on Friday May 09, 2014 @05:49AM (#7720087)
Perhaps it'd be less bad if Windows wasn't so gratuitously different.

On a more serious note, there's a pretty large gulf between the Windows and Unix development worlds. Try getting a game developer to use curl ...

Re:Searching 20 GB/sec: Systems Engineering Before Algorithms by shantanugoel on Monday May 12, 2014 @02:49PM (#7734648)
Aside to the discussion, can you point out a link to the changes you did? I'd like to get silver searcher faster as well on windows.
Re:Searching 20 GB/sec: Systems Engineering Before Algorithms by to3m on Wednesday May 14, 2014 @07:23PM (#7747106)
Sure. Here's a binary: https://github.com/tom-seddon/the_silver_searcher/tree/_vs20...

The source code is there too, but I wouldn't look too closely. "Cobbled together", like I said. Though I've been using it all the time, in conjunction with ag-mode - https://github.com/Wilfred/ag.el - and haven't noticed any obvious problems.