As someone also hosting thousands of songs who has been using Navidrome for months I can't say I experienced any performance issues. The codebase is fairly small so it should be fairly straight forward to investigate and narrow down the performance issues you're experiencing.