The solution is simple: always speak English. This fixes other similar problems.

The other problem with their voice recognition is speaking English with an accent that isn't US or RP.