> If it was actually perfect grammatical accuracy on their part they’d be saying “Who were you talking to” or “Who were you speaking to”.

There is a sizable contingent of English speakers out there that continue to insist that it’s incorrect to end a sentence in a preposition. (I am not part of this group and it is fortunately shrinking over time.)

Hence Churchill's response to said contingent that "This is the type of arrant pedantry up with which I will not put."

There's an old joke:

  Texan: Where are you from?
  
  Harvard Student: I'm from a place where we don't end sentences with prepositions.
  
  Texan: OK. Where are you from, jackass?

> There is a sizable contingent of English speakers out there that continue to insist that it’s incorrect to end a sentence in a preposition

That is the sort of nonsense up with which one should not put.

“With whom were you speaking” IS grammatically correct though. “Whom” is the object of the preposition “with”.

Even if you want to differentiate "who" and "whom", "Whom were you speaking with?" is more natural.

I use English according to defined grammar rules. It marks me as upper middle class and therefore trustworthy in administrative professional positions, greatly improving my life outcomes compared to someone who is not able to do so.