I wrote a similar piece of software a few years ago. I solved that by having a config flag for each clipboard agent to disable broadcasting of their clipboard changes and only listening for changes. Then I simply activated that on the box that holds keys/passwords etc.