SPA applications support the ability to update parts of a web page e.g. an article content might be 5K out of a 20K file. A server side website has to bring down the 20K page again and again as the user navigates through your site. Caching or web server configuration makes no difference here.
Browser caching, not server-side caching. The browser doesn't have to "bring down" the 20k page "again and again", it gets the assets out of its cache (html, js, css, images, etc), and then the JavaScript loads the 5k article when the DOM is ready. Hope you're not trolling. Non-SPA websites can be basically just as fast as SPA's, and in practice they are often faster since they are usually built without frameworks and have less JavaScripts.
We aren't talking about CSS, JS, Images but purely about the HTML page. If you have a 20K page and 15K of that belongs to header, footer, sidebar then in a server side rendered site that 15K has to be downloaded for every page on the site. On a SPA site it only has to be downloaded once as you can dynamically switch out the other parts.
If there was a mechanism in the HTML spec to tag parts of a page for partial caching then we wouldn't need SPA in most cases. But browsers only cache the entire HTML.
Not true. If you do proper caching and configure your nginx properly back-end rendering can be just as fast.