If our experience itself isn't what we think it is, that would tend to undermine empiricism itself, because we rely on our senses to inform us about the natural world. This includes our sense of whether or not others agree with us, whether they have the same experiences as us, etc.