They waste time and lower information density, and also, for the few people who do notice them, they get very irritating. Getting rid of filler words where natural is pretty standard practice for longer-form videos.

If you're using a video to obtain information, you're already resolved to a low information density.

Jumpcut editing - popular amongst some of the youtube crowd - can be way more frustrating for some consumers than hearing a person periodically pause with an uhm / ahh, or insert the word 'like' several times a sentence, or start every sentence with 'So...'.

I’ve been receiving scam voicemails that use some AI to try and sound like a human that adds umms and uhg’s to try and sound more lifelike.

Often when people are aware they're being recorded, especially for some kind of production like a podcast that they're accustomed to hearing with edits, they'll make far more umms and ahhs than they would in an ordinary conversation. I think maybe its because they're concentrating on the continuity of their speech? Whatever the reason, I usually just try to get the number of these tics down to a level that sounds like casual conversation. I don't think most listeners notice when the tics are edited out, but a significant number seem to find them annoying enough that some judicious editing improves overall happiness with the production :)

because in my audios I look (hear?) smarter without it :-)