Realistically, I will be reviewing a diff, see a hunk I don’t like, change it, and want to see the change back in less. So saving my scroll offset would accomplish my goal. I guess I should add that I want to be able to quit as well?

Like I said, fzf does this. Bind a key to an action that effectively changes the file from stdin to a different command that it runs, while preserving view state.