Thinking of it as 'I' or 'you' is misleading. That's kinda what I'm trying to say in my post: it's more like these are all third-person roles. If you think of 'boku' as more like 'this boy' than 'I' or 'me' it becomes easier to understand I think.