This idea that unwelcome advances is harassment is ridiculous (not saying that's what you're saying in your comment, just what I've heard countless times in discussions).

Asking for a phone number is ok at almost any time.

If you ask me mid workout I'll be annoyed though and likely react negatively.

If you ask me and I'm engaged / not looking, it will be annoying. Some women train in women only gym for this reason. I still think it's ok to ask in a polite way.

Just don't ask in a situation where the receiver may feel threatened (eg. Alone in a dark alley) and react out of fear.