In my example, the "them" isn't referring to the recipients of the letters. It's referring to the letters themselves.
Mallory: "The mill received a series of letters. Those letters are evidence. Now, we're not leaving here until I find out who wrote those letters. Alice, did you write them? Bob, did you?"
Carol: "I wrote them."
"There was a problem at the mill so I wrote them" - this sounds wrong
"The mill workers were complaining so I wrote them a letter" - this is fine
"The mill workers were complaining so I wrote to them" - this is also fine