There are answers to this. I asked myself this a few years ago, which led recursively to new tricky questions. I eventually solved the tree of the questions: I designed myself into a solution that I later realized was essentially an asymmetric PAKE (e.g., analogous to OPAQUE).

So, an attempt to solve your question eventually leads to using an asymetric PAKE. But, this is way, way more complex. You'd really have to squint your eyes to believe that the diminishing returns are worth it.