But I grant you, it has been a very complex, painful journey to get this working right, and in fact I'm still making tweaks and adjustments. Plus, often times it's really unclear if I should scale vertically or horizontally.
TBO, it does not sound like it's working great for your use-case =) Wouldn't something like Cloud Run, AppEgine Standard or Heroku be much simpler and cheaper? Or just a single $5-15 per month* VM?
* depending on how much oomph you need for those 400 rps
I guess the ultimate goal of services like GKE autopilot is for you not to have to worry about kubernetes at all, just give them your workload and have it run on whatever resources they think is appropriate?
I do think it's important to recognise though that there are lots of ways to host services, and simpler with less abstractions is often better and more reliable and certainly easier to debug when things go wrong.