I mean classical music you follow the conductor, even as they bend the time faster or slower, or take unmarked pauses.

The notation is a starting point to learn the baseline notes and rhythms. The actual end product is never going to actually match up 100% to the written version except maybe in like middle school.