In my experience in trying to learn this stuff, I have to say, I can't say the young people around me are trying to do the same. I think they're of the mindset where they just want to solve their problem, and whatever gets it done with the least hassle is 'good enough.' The truth of the matter is that they're generally right. Computing resources are so expansive that for a lot of problems are of small enough n, that people ignorant of data structures and algorithms get by just fine. Just using decent DS & algorithms will get people the speed boost they need most of the time in the higher level languages. If you're really in a pickle, you break into C.
When I was working as a web developer, I never saw someone other than myself actually get down to the C level. They just bought more machines and scaled that way. This stuff is fast becoming something of a lost art, but there will always be domains where it is necessary. I happen to think that's an interesting place, but the trend in development is well away from it in most other domains as it climbs up the abstraction ladder.
EDIT: I'll just add that I've seen the ignorance of these things (just at the algo level even) leave money on the table. I think people worship at the table of 'cycles are cheap' a bit too much sometimes and discount hardware and maintenance costs a little too readily. It's a balancing act.
That's one for the fortune file.
And in short, our target CPUs don't have SSE. I've only been vaguely aware of SSE, although I used to program a Cray on occasion, so am familiar with vector processing. I must see if the SSE instructions can accelerate some of our other work which isn't yet critical, but may become so.
The demand driven stuff was done by taking 32x32 tiles in the target and reverse transforming each to the source, hence finding out which 32x32 bit tiles in the the source needed to be computed. Flags prevented redundant multiple conversions. The transformation doesn't preserve straight lines, so some care was required on boundaries.
From reading a little bit of T90 documentation, SSE looks pretty different from the Cray vector stuff. It's little more than the kind of (x & (!x + 0x01010101)) kind of stuff you're talking about above, except with somewhat larger registers and a wider variety of instructions.
Actually, most of what I did are standard techniques that assembly programmers know, C programmers have seen, and web programmers don't need to bother with.
I can't actually post code here because it's commercial in confidence, yada yada yada, but I will mention the techniques.
Firstly, they were processing an entire image, then transforming it, then getting the part they wanted. We tranformed the program flow to be demand-driven, which meant that large amounts of the image weren't processed at all because they weren't needed.
Then we analysed the data and deduced that by taking logs we only needed a dynamic range of 6 bits. We therefore took the entire image into a byte map with the top two bits of each byte set to 0.
Processing 4 bytes at a time in words saved enormous amounts of memory access, and avoided register spills. Unrolling loops by the right amount meant that we stayed in cache, but had less overhead.
Once the processing had been re-worked, we looked at the assembly produced, and found that most of the time the processing could be separated into a double indirection.
Finally, by further reducing the resolution of the system, computing an approximation, then reworking those parts of the data where the errors were too large we managed to halve the work being done.
It was a good two days of analysis, and the code shrank by a factor of 2, but it's now tough to understand. We have documented the results. It's embedded work, and yes, we finally went to assembly to get the last 30% of performance.
Every part of it is standard in its field. I think some of my programmers hadn't created routines to work byte-wise eight bytes at a time in a 64 bit word, and others hadn't done the transform via logs to throw away unnecessary resolution. Another had never analysed unrolling a loop as compared with staying in cache. We used tricks such as (x & -x) to find the least bit of a number, although in 32-bit space that becomes something like (x & (!x + 0x01010101)).
And so on. A large collection of esoteric tricks, all at once.
Let me finish by saying that programs can't be made to run faster, they can only be made to do less work. An interesting aphorism when "optimised" code.