The problems was all my fault :-) I was trying to use a port that was not designed for serial data. When data was sent across it was getting mashed. I think it's because both ports were not uart, therefore when the binary data was sent if they were not perfectly in sync it would get mashed, I might have been able to solve it by sending a clock as well. But the easier option was to just change everything to the uart ports and it magically worked.