Interesting - from this and a couple of other comments I can see my idea that you just had to pay US income tax regardless was over-simplified and wrong.

It's still a little weird to me though, as someone who would expect my own country to relinquish all tax demands if I'm not actually there!