As Jeff Atwood famously puts it "performance is a feature", so as any with feature, wether you should invest in it depends on the project and the needs. Sometimes it's worth your time sometimes it's not, only your specs can tell.