Text is usually made up of lots of small solid shapes whereas SVG will usually have larger shapes, sometimes using effects such as gradients - so it's likely that different optimisations will work best on each one.