Still counts as a win, you can IME take out a cache mechanism easily. If somebody wants to ding you for writing excess code I don't know what to tell you, they are probably a-holes.
Postponing optimization is fine, but if everyone is just postponing it, and nobody is treating it as priority, then it just never gets done. In most software, low performance is effectively an externality - it's unlikely to be the thing that kills your product/business[1], but it's causing real frustration in people and waste real money in the overall economy.
----
[0] - "Work expands so as to fill the time available for its completion"; https://en.wikipedia.org/wiki/Parkinson%27s_law.
[1] - Or at least not obviously so. Performance issues often masquerade under general complaints of bad UX, or the software feeling "off" somehow. Also performance is often a limiting factor in use cases; it can be what enables you to access a whole new market segment, or deny it to you.