I agree.
Unfortunately, in Saigon - we get a lot of the BroPackers and they tend to all convene in the Backpacker Area, Bui Vien.
I also have local friends.
I did a motorbike trip with my local friend to the Southern Vietnamese city of My Tho in the Mekong Delta a few weeks ago.
Quite normal to have local friends.
How exactly do you integrate with Vietnamese culture when you don't speak the language or have any other connection to it? You see this stupid argument on the expat groups on Facebook all the time, people slagging off on expats that live in the more western districts because somehow living in a western style apartment isn't "authentic Vietnamese living". It's a giant pile of bullshit, imho.
If you mean "have local friends" or an integrated social circle, almost everyone living here long term has that. You can't live here for a long time and not have it.
Maybe you got stuck in the teachers on paid vacation circle, but everyone I know here doing stuff outside of that doesn't even remotely fit your description.