Yes.
One should avoid to extract a component from the repo as a submodule too early, i.e when the interface of the component is not mostly stabilized.
If each commit in your "main" module is paired with a commit in your submodules, then you might be missing the point of module separation.
Even if the interface is stabilized, when you update implementation of a submodule you will still have to recursively update all submodules from main down to the changed module.
My own positive experience with submodules involves dependencies that only occasionally change; I can always point to a specific tag that should be used.