In my example, the "them" isn't referring to the recipients of the letters. It's referring to the letters themselves.

Mallory: "The mill received a series of letters. Those letters are evidence. Now, we're not leaving here until I find out who wrote those letters. Alice, did you write them? Bob, did you?"

Carol: "I wrote them."