If you’re not tailgating someone, you’ll be fine, and anyway “the rest of us drivers” includes plenty of people who drive much more dangerously than an overly cautious AI. Autopilot has many millions of miles of driving time and as far as I know there hasn’t been even a single fatal accident due to phantom braking.