Traffic lights, crime-addled regions, embassies (I wouldn't dare jog towards a guarded embassy carrying a backpack), loose gravel paths. All these will slow down walking considerably.
Or outside the city: Marshes, mountains, thick forests, muddy hill slopes.
Same with your car: the places without roads are undrivable and therefore not available for routing. But with a car, the speeds are so much greater than bipedal motion, shortest vs. fastest is a viable question. When the routing system avoids obstacles, it's mostly going give you the shortest and fastest route. brador makes a good point about grade; with enough grade on the shortest route, it might not be the fastest.
It's a quick and dirty calculation that google is making about routing. Im pretty sure the google walking routes wont take you through marshes or thick forests.
Or you could analyse ground gradiants for steepness and find the flatest. Combine with your own personal fitness levels and solve for a better optimal route :)