I say the refusal to answer is an answer; since you are asking the very specific question that reveals if you're under custodial interrogation, a refusal to say "Yes" has to mean "No". I mean, unless you get that yes, you clearly can't leave, right?

Furthermore, one could hope that your refusal to answer his questions with anything other than "Am I free to go?" would not be so easy to construe as this sort of silence signaling guilt.

I'd take it the other way. If you're not under custodial interrogation, they have no obligation to tell you anything - so a non-response to "am I free to go?" amounts to "yes". I mean, unless I get that no, they're just stalling for time to find some reason to detain where they otherwise have none. I'd be awfully tempted to include a "if you don't answer the question in 30 seconds I'll assume the answer is 'yes' and I will leave", but methinks that just complicates things. If they can't affirm arrest (however loosely defined), their delaying you is NOT for your benefit.

Ah, but they are still spending time on you, being there in your face. If they truly weren't interested in you, then they'd leave in search of easier prey.

Since your leaving could be spun as felony (in at least some jurisdictions) fleeing the police, you'd best not leave without their OK, explicit, or implicit by their leaving first.

Note: don't let yourself get startled by the police into running away. One of the things that reified all I've been reading about the modern police in the last decade or so was retiring and returning to my home town, and one day in 2007-8 a new, improved policeman played a game of chicken with my body and his car, that is, as soon as he noticed me, he veered it on a near collision course. I was an experienced enough pedestrian---which I think is unusual here---that I could tell his vehicle would miss me by a good foot or more and the mirror by at least half that, so I stared him down, and he then veered away. Based on my readings, I have have to assume he was trying to get me to run ("this crazy cop is trying to kill me!"), at which point the crime of fleeing the police could be claimed.

They're checking everyone, and do not take well to anyone who questions their authority with "you have no reason to detain me, so I'll be going now."

And your point is?

Nowhere am I advocating that. I'm only suggesting "Am I free to leave?" and silence in response to anything but an unequivocal "Yes" to the question. And noting their leaving is also an implicit Yes.

Now, depending on your reading of the situation and the cop(s), you could play games with an equivocal answer, i.e. follow with "OK, you are detaining me", and if they object, go back to the free to leave theme, then press them on deciding one or the other. But that's not what I think one should generally do.