If I have a single page web app, I don't want the overhead of returning markup. I just want JSON. And when I get raw data back instead of a string of HTML, I can be a lot smarter about responding to user input (e.g. optimistic updates).
Also, sending JSON back and forth is kinda nice because it's fairly compact, easy to handle on both sides, and additionally this particular API may be used for other purposes, too.
Rendering speed isn't the reason server-side templates feel "slow" though: if you do server-side rendering then every change of state on the client needs to round-trip to the server in order to fetch a new HTML representation. Client-side rendering lets you "cheat": just render the new state and synchronize with the server after the fact.