Concrete without rebar lacks tensile strength, and Roman concrete is no different. When using materials with insufficient tensile strength, you're severely restricted in the kinds of structures you can build. That's why Roman buildings used so many domes and arches, and why they were massive.

Practical Engineering has a good video about it: https://www.youtube.com/watch?v=qL0BB2PRY7k