You do need to know where to send the user's return traffic, so you'll need a table ultimately comprising mappings of network flows to end-user addresses. Of course, once the flows close you don't need to retain this information. In practice, you'll also need information about all currently-open VPN sessions.