It's not so much the registers themselves, it's the plumbing required if you're going to let any register supply any input or output of the ALU. Hardwiring a register to one of the ALU's ports means you avoid a multiplexer (and the same applies to hardwiring a register to the AGU's ports, which is how you get 'base' and 'index' type registers).

Sure, there are lots and lots of deeper details but fundamentally, more registers means more transistors (or a higher-level logical unit of choice) and it's the constraint that drives accumulator-centric designs, specialized registers, etc. Relax the constraint and they become less necessary.