I experience this as well. I remember that in school, I would always seek out the most remote corners of the library in order to work. And before Covid times, working in an office that was anything but dead silent was a constant source of "activation" and therefore subtle distraction.

Working from home has posted its own distraction challenges, but I do not miss that "nonfunctionally distracted" feeling I often got in the office.