R&D doesn't require something to be socially innovative; it's evaluated at a per-company level. It's intended to incentivize companies to recreate wheels in-house (if it is otherwise financially reasonable to do so).
It does not create a barrier to small companies. If you're not big enough to handle the paperwork for the R&D credit, you're not spending enough to derive meaningful benefit from the credit anyways. And generally, there exist plenty of accounting firms that will prepare the R&D documentation on a % basis (of the allowable R&D credit identified).
It also subsidises product development engineers - a profession that is doing well already
In the software world, sure. There are plenty of industries where product development staff (many of whom aren't engineers) don't make anywhere close to 6 figures.
From what I've seen, many companies will shoehorn regular product development into the R&D category. I'm not sure if it's supposed to cover next iteration of existing products that most company should be doing anyway.