I am not a lawyer, but my understanding is the same as yours - if you host a proprietary service that has some AGPL components (say, an AGPL authentication service that users interact directly with to get a token, that they then pass to the proprietary parts), then you only have to make sure that the source code for the AGPL part is made available (or not even that if you are using it completely unmodified).