Those fixed 96 (or whatever) steps of processing limit the complexity of what the model can do, so it will fail if the task is too complicated unless it it breaks it down into simpler steps that each can be done well with that depth (96 steps) of processing.
It's not just appearing to do so - with chain-of-thought prompting you are literally telling it to "think step by step" as part of the prompt, so this is what it outputs. You could also tell it to generate a step by step plan, then elaborate on each of those steps.
I don't think we can say exactly how it is deciding to break a task into steps, anymore than we can in general say exactly how these LLMs are working, but intuitively it's similar to how we think and talk (which is what the LLM is trained on) - a good speaker/writer will introduce a complex topic as a top-down decomposition.
Because the model can't "think". There is no "reasoning" that goes into generating an answer. So for complex problems that require multiple steps of reasoning, the model needs to persist those intermediate steps in order to be able to build up to a solution.
If you look something up (on the internet, in a book, asking people directly) and receive two answers, one which describes the steps used to arrive at the answer, and one which doesn't, on average which one is more likely to be the correct answer?
As a prediction machine, an LLM is more constrained by what is likely to appear after a chain of reasoning.
Sure, but why does that make the model more effective? Are you sure it's "breaking the problem down into simpler steps", or is it just appearing to do so? How does this breakdown happen in the model, exactly? If we can better understand the mechanics involved, then maybe this process can be built into a new model that can achieve the same results more efficiently instead of as a recursive process that runs the model more than once.