Let's see. As these early ML-heavy prototypes will start becoming industrial and safety-critical products, the emphasis will be on explainability and graceful degradation. This inevitably leads to a hybrid system with poorly vectorizable parts (expert systems, ad-hoc solutions), which will have to be tightly integrated. Also, full of various auditing and monitoring systems, which are "oligo-threaded" by nature.

Therefore you'll need a capable several-core CPU anyway, and unless the memory is shared (which means you have a SoC, where there's no clear leader AFAIU), moving data in and out becomes a major bottleneck.

I think for most systems where explainability and audit trail are important it will be easier (I mean cheaper) to build on a powerful CPU from the start, and when the system is successful (i.e. approved by real customers), then the highly-parallel parts may be factored out to special hardware. Using them from the start is very risky.

These thoughts are from experience building an all-weather computer vision system for traffic control applications (99.5+% vehicle classification, with errors requiring human intervention and explanation), and being rather familiar with a company which makes specialized ID-recognizing mobile apps.