In that case I would do whatever was necessary to get that required performance - but only after doing extensive performance testing to ensure that's really where the bottleneck is and providing copious comments and possibly supporting documentation to justify this approach.

99.9% of the time I have seen code do crazy things "because it is faster" it's not performance critical code anyway and there is no explanation provided.