If he's pulls over and asks you where you're going, then he's not actually off-duty. If he refuses to take you (either because you don't answer or because he doesn't like your answer) then you should report him.

Authentically off-duty cab drivers will do exactly what you say. They ask you where you are going and don't unlock unless they like your answer. I'm not saying that abuse never happens, but this scenario is perfectly valid for cab drivers on their way home.

Source: http://nycitycab.com/NYC%20Taxi%20Guide.aspx#offduty

The lights.

A NYC cab's lights look like this:

|O|Number|O|

Where "O" is "Off Duty" in smaller letters.

If just the Number is lit, they're available, on the clock.

If Just the Os are lit, they have a fare.

If both are lit, they're available, with destination discretion.

If none are lit, they're not working at all.