It was probably due to the lack of flow control. Serial doesn't work well when it's one-way. I did something similar to send logs waaaay back in the day, and it would constantly flip bits or send characters out of sequence, etc. I had to transmit very slowly to get it to work stably without any flow control. I want to say that I limited it to 9600 kbps before it started to become reliable.