But if you do not take care the "more important things" will have to be refactored or even rewritten allo over because you build them on a poorly performing base.
Finding the bottlenecks should always be a priority. Fixing them can be done at a later point.
Putting effort into performance is probably something you'll want to do at some stage if your app gets big enough. But a lot of the time you have more important things to do right now. Users will put up with an awful lot if you've got something they really want; get that part down first.