Architecture-wise, even a regular modern CPU isn't that hard. For example, Berkeley has the BOOMv3 core[0] which is performance-competitive[1] with commercial chips taped out in the last few years. I think commercial chips are faster because of improvements in analog design, and not some super special architecture sauce (although I'm sure there's some special sauce -- it's probably not the defining factor).

[0]: https://boom-core.org

[1]: https://carrv.github.io/2020/papers/CARRV2020_paper_15_Zhao....