In that case get the two others with you and decide a new rule: nobody leaves the retrospective before they've spoken.
Warning: might backfire, one team I worked with scheduled meetings right before lunch so they wouldn't stretch out.
It backfired: these meetings continued into our lunch every single time until I think I or someone else started just walking out as lunch started.
More likely you have people who are paid to do some workload which they are generally happy with. They aren't going to be motivated into doing more by accident, especially through their own action. This is the vast majority of people in the world, so good luck countering that momentum over the long term.
Effective software project management is an interesting and hard problem, granted, so many people are interested in how to tackle it. I get it.
What I see is that "agile" criticisms are really criticisms of it's inefficacy in giving employers more for their dollar from the employee perspective. I don't see why I should get too invested in minimizing my relative value.
If you have a bunch of people who are just doing the minimum they’re paid to do, with no passion and excitement for the work itself or their team, you either need a different method or different people.
It’s like shaping a system of people. I think of it as 50% of the development endeavor at larger shops.
Retros are used to identify ”technical organization debt” starting to build up, in a way.
In fact, all agile needs is good retrospectives, and all the rest you can decide there.