I think I used the word vouch incorrectly. When you sign a commit you indicate that your changes can be applied only on that particular parent even if all files or diffs were bit for bit identical. If it didn't work like that someone could take your work and place it in a completely different project and claim it was signed by you.

Note that even with signed commits there are same gaps in the system. For example without signed pushes someone can take your signed commit from branch "test" an push it to "master".