This mirrors my experience learning German, in Germany, pretty much exactly. I falsely assumed I would just somehow pickup German by being exposed to it every day but the brain is amazing at filtering things out. I'm attending German lessons 3 evenings a week now but it's still a rough road.