Mask ROM usually is just used as a big lookup table which replaces the combinatorial logic by just explicitly matching each input bitstring to the correct bitstring to control the ALU, etc. (along with some sort of state-machine for VLIW ISAs).
Moving to a ROM-style architecture makes development a little more straightforward because modifying instruction decoder only requires burning a different bitstring into a PROM rather than redesigning a combinatorial logic circuit and then manufacturing an entirely new chip.
A ROM-based architecture probably makes microcode updates easier to design into a CPU as well, because you can basically just overlay the microcode over the existing ROM and patch, at runtime, essentially any part of the instruction decode (or whatever else you've designed in the ROM) on an unmodified production IC.
I imagine that the combinatorial logic approach isn't really used anywhere today (for new designs) because of how "cheap" gates are considered to be in a digital IC design, but I could be entirely wrong about this (I'm not in the industry).
What's being discussed is at the same level as microcode, whereas most of the distinctions you're making apply to the normal code storage ROMs. For instance modern microcode is nearly never a PROM.
Additionally combinatorial logic is generally made out of cheap gates, the two concepts aren't in conflict, but instead orthogonal.
My understanding (as explained to me by someone who did work on some CPU-like ASICs) was that they were able to produce the early versions of ICs with some form of "PROM" (possibly not the correct term) which could then be one-time programmed so that they didn't have to pay for a new set of masks if they wanted to change any of their instruction decode (or whatever they had wired into the "PROM") logic.
All they had to do was burn a new version into a blank chip and test/use it from there.
The company in question was producing low-volume, high cost products for a narrow niche and so I don't know if they had to throw away some die space to have this "PROM-like" feature and if that is still how things are done today.
I my comment earlier about EDA tools being able to "simplify" the ROM into the die was probably a paraphrasing of what I was told about a tool. The tool (or workflow) allowed a designer to go from a working prototype microcode ROM to a hardcoded gate circuit (presumably by the same sort of process that one can go through to simplify a logic statement that is in disjunctive-normal form).
This was information passed on to me second hand from quite a few years back now so I probably should've tried to look up some primary sources before writing my comment! Oh well.
For instance the opcode 0x76 should be "LD (HL),(HL)", but instead this opcode has been reused for the HALT instruction.
The Z80 has undocumented instructions too (for instance in the prefix-ranges for the IX and IY indexed instructions), but those behave the same as documented instructions, except that they cost more cycles. This seems to indicate that the instruction decoding on the Z80 works quite a bit different than on the 6502.
The 6502 approach of just executing some random (but deterministic) logic leads to a situation where the "illegal" opcodes cannot be expected to be unused (programs actually use them), and where assigning them new semantics effectively breaks compatibility with existing programs. That’s why CPUs usually invest in the extra logic necessary to give those opcodes undesirable behavior (i.e. traps).
It preceded the Z80 by a year or two.
Something I found especially interesting is the decision to not implement "STX abs,Y" and "STY abs,X", while the decoding matrix would have easily provided for this. Was this to keep the instruction set arbitrarily simple (as in explicitly not orthogonal)? Or was there a more serious architectural conflict? Or was this just a legacy from the 6800, which had just a single index register, thus lacking a similar option?
(I added a note on this to the article.)
This [1] summarizes it. Here [2] is an interview by Computer History Museum of Chuck Peddle. It matches up with the first chapters in Bagnalls book very closely. From about 50 minutes in he talks about his meetings w/Motorola customers that told him the 6800 was too expensive. From about 58m in they talk about basically setting a size metric for the chip to meet a low enough price point. So the starting point was what they could fit in small enough space.
You might also like the Computer History Museum interview w/Bill Mensch [3]
[1] http://www.cpushack.com/2013/08/03/mos-technology-mcs6501-pr...
https://www.pagetable.com/?p=39
...it's basically a tolerated side effect of the very simple table-based instruction decoding on the 6502. The main design goal of the 6502 was to create a very cheap yet practical CPU (compared to more complex and more expensive CPUs like the Z80), and one way to achieve this was to simplify the instruction decoding with a simple "switch board" even if this means that there's nonsensical "cross-talk" for some opcode bit patterns.
Here's also a very nice blog post by Ken Shirriff about the instruction decoding differences on ARM1, 6502 and Z80:
https://www.righto.com/2016/02/reverse-engineering-arm1-inst...