I practise the 'path' method of speed optimisation: If you can not execute a piece of heavy code every 10 in 100 runs, you save a lot, just by a simple data/value check before it.

As for memory opts, pass by reference instead of value (in C++).

I've sped up code by a factor of hundreds by doing some crazy stuff (without going to assembly).