A lot in this area comes from an expecience in operations or at least operations mindset.

If you haven't solved certain problems caused by real users and real loads on production systems, you can still develop such mindset.

Build a habit of overloading your app with tons of requests - using load testing tools like wrk, wrk2, k6. Try to check this way all layers of your system, including the database (for this you obviously need to be able to create load-testing requests causing a real conversation with the database). Check how much of i/o, memory, and cpu (in this order) your app, database, cache, messaging etc is using under the said load. Develop a habit of opening inspector in your browser and peek at http conversations, not only for your services, but for a random page you visit. Get acquanted with Brendan Gregg's perf tools and his blog. Know how to use strace (although it's usage on production is discouraged as it slows down the examined process substantially) and understand its output. In general, become interested when your code ends up doing kernel syscalls and when not and, how to limit syscalls by buffering and how to read what's really going on. Make sure you understand how the virtual memory works on a modern system. That's just things from the top of my head