I believe this is a point of contention among street photographers. Some believe you should always ask permission, others believe that if you are in public you are fair game.

Also laws may vary depending upon where you are.