The main drawback is really when you attempt to do more advanced prompting techniques like chain-of-thought or reasoning.

forcing those parts to be json, can be hard and unnecessarily constrain the model. e.g. https://www.promptfiddle.com/Chain-of-Thought-KcSBh

try pressing run tests and you'll see what i mean! this method or doing chain of thought works a bit better