Just swim down when you are grabbed. If you kick and thrash to try to raise yourself to get your head above the water, they will feel you raising them up and try to climb on top of you for safety. So you shovel water with your arms to push yourself farther down, pulling them down just enough that their heads go underwater, too, and they'll release you quickly. They are struggling desperately to keep their heads above water and won't hang on to something that's pulling them under. Your head is already underwater, so it won't hurt you to go a little deeper, but if you panic and try to fight toward the surface, they might hold you under until it's too late for you. Go down.
Once they release, you push back out of reach, get your air at the surface, and figure out how to help them.
You're right though, the safety of the rescuer always comes first.
I know Red Cross teaches releases, I think EA might have a small paragraph in the book devoted to it, but a large section on the importance of the lifeguard tube.
I've saved > 10 kids over the years I guarded and never had an issue with grabby victims, but 90% of my victims were kids in 5ft of water. I did have a coworker get grabbed in a 12ft diving well once though. Thankfully she was able to get free.
The worst part was we were supposed to look the other way and have them come up behind us and tackle us in the water. The worst part was having a 300 lbs old guy doing who isn't a fan of making it easy.
I never gave much thought to the effect this might have in open water if the drowner did not see you approaching - suddenly something underwater is grabbing him and controlling his movement - Jaws!8-))
Edit: actually perhaps not the best idea.
First rule of rescue: your own safety first. Not only does that minimise the casualties, but if you're injured or dead you can't help others anyway
Unless you're a trained lifeguard this is poor advice. As stated above you don't want to get within touching/grabbing distance of a drowning person. Get some sort of intermediary between you and them, a stick, rope, shirt, pants whatever. Something that you can drop if you are at risk.
To the point below about their instincts of not grabbing it. Don't leave it out where they need to reach for it. Hit them in the head or arms with the stick or piece of clothing and they'll grab it. My first few pool rescues with a lifebouy I'd leave it within their grasp and they'd not grab it. If I hit them in the head or hands with it they'd grab it every time.