I've been on the other end of that story, where entire dev organizations freak out because of a concern with webpack, babel, or some other piece of the stack... and nobody knows how to fix it.

CRA has its place, certainly. But it is a black box, and somebody in your org should know the tools well enough to make good decisions to build up a product from scratch without CRA... and then you may still choose to use CRA, but it is an educated choice, not a default answer because you do not know any other options.