Uh, there is a private fork in this scenario. The whole point of AGPL is that you cannot legally modify the software and host it over a network to avoid having to publish your fork and comply with the GPL copyleft clause. Mere use does not trip the AGPL copyleft, because Freedom Zero is considered sacrosanct.

Furthermore, there are plenty of situations involving a network, other than SaaS, in which both the client and server could be considered end-users. For example, a WordPress site. The people who own the site and the people viewing it are two separate sets of end-users with their own freedoms to protect.