Which the client might be responsible for because the contractor might not be allowed to retain the IP

You're "allowed". Just keep it to yourself.

You often will not be, explicitly by contract. Honestly in this case it’s easier to keep things clean.

If you do some work for someone one and decide to keep the source you wrote, so you can review it later in the privacy of your own thoughts, the only way you can get "caught" is if you release it some way that's recognizable.

I wouldn't want to hire anyone that thought like this, as it demonstrates a lack of morality and willingness to adhere to not just an informal agreement, but a legal contract that could expose me to liability or IP theft.

If that's what you want to do, why would you ever sign a contract saying you won't do it?

If the work is interesting and well-compensated, sure, I would sign that contract.

I will also keep the source I write and, if I feel like it, take a look at it years later and no one will know except me. BFD!