Authentically off-duty cab drivers will do exactly what you say. They ask you where you are going and don't unlock unless they like your answer. I'm not saying that abuse never happens, but this scenario is perfectly valid for cab drivers on their way home.

Source: http://nycitycab.com/NYC%20Taxi%20Guide.aspx#offduty