That is a slight concern, but I don't see it happening, at least in Australia for the big four banks, in the near future.

If that became the case, then the 'glorified token device' would become the dedicated banking device, and not much else would change (ie. I still wouldn't be doing 'banking' while I'm out and about).