The root of the problem is that I2C has serious EMC immunity issues. It's well known and appreciated that its drive is weak and open-drain. The bus buffers and especially differential drivers can and do help there. (And will get you past a meter.) What's less well recognized is that a single glitch pulse on SCK knocks all the internal state machines out of whack and requires a bus reset to fix them. Hope you're doing that when your bus is idle or when you're getting anomalies! Most people don't. The Nexus 4 phones sure didn't; this is why their light sensors went dead or crazy or both after a while of uptime.
All of that gets easier to handle if there's a nice, big, low-impedance ground plane nearby, which is why you don't see so much trouble when it stays on the PCB.