This is not the set of flags you have in the makefile on github though. I would expect people to use those :)

In any case, building warp with this brings the timings down to 2.31 seconds, so clang is still 40% faster (1.41 vs 2.31)

In any case, at least on my side, i don't have time to further explore, i'd love to see cases where warp is faster, but i haven't found them.

(There is also a certain irony of saying you don't post numbers because you get accused, then saying i used the wrong flags, but ...)

Thanks for doing this. I have read that clang uses some SIMD instructions to speed this up, and I don't know how much that contributes. Warp doesn't use any inline assembler.

And, as your numbers show, suggesting the change in compiler flags was entirely justified.

the SIMD usage is mainly to do two things.

In the lexer, it is used for block comment skipping. It will find the end of block comments 16 characters at a time (on both PPC and x86).

During line number computation, it will also find newlines 16 characters at a time.

This could actually (nowadays) be done 32 characters at a time on newer processors, but isn't.

This is flat-out fascinating. Thanks.

I'm curious how much of an effect this has.

64 characters with AVX-512? =)

I didn't check to see if the instructions exist, but possibly :)

You do start to hit two issues though as oyu increase the size of the skipping:

1. Alignment 2. If the average block comment/line is < 64 characters, you may lose more time performing the instruction and then counting the trailing zeros in the result to find the place it ended.

I have no numbers to back up whether this matters, of course :)

AVX-512 does not seem to have PMOVMSKB, which is how I assume it is being done with SSE2. There are other ways to skin that cat, but it's unclear whether they have any advantage over using AVX2 with VPMOVMSKB.

I posted a patch here: https://gist.github.com/dberlin/9867614

It adds AVX2 and SSE4.2 instruction support. It makes no discernible difference performance wise that i can find :)

Heh, awesome! I'll try it out today.

I forgot to ask - are you compiling Warp for a 64 bit executable? Use -m64 if you're not sure.