It's not even really a black box. It's learned a subset of Abaqus's functionality so it's scope is known.

Edit: Further elaboration from the article: product designers, for example, could test the viability of their ideas before passing the project along to an engineering team

This would seem to clearly indicate that the intended application is faster iteration in design, not a replacement for a rigorous engineering process that would include appropriate testing.

Estimating

It might even be good to analyze inspection photos for regular inspections to draw attention to situations where there might be more strain than anticipated and a human could be flagged to inspect more closely.

In my experience building and working with such models in similar domains, they tend to be right in all the easy and uninteresting cases where there are no problems and in all the cases where the problem is super obvious, and wrong in all the hard and subtle corner cases that you actually care about.

The other problem is that traditional engineers hate models they don't understand. If you can't explain, at least roughly, why it gave the answer it gave, then they won't trust it and won't use it.