That's not an issue if there's supply. What you're describing (unsubstantiated) is a lack of supply being exploited.

Realistically we cannot build enough supply for distribution not to matter.

Why?

Is inequality bad in and of itself? If there's a healthy market of buyers and sellers, but also a lot of inequality, is that a problem?

Yes, it is bad. The distribution of wealth determines the distribution of resources. The housing stock is limited. It ends up being mainly owned by those with a lot of money, in the process putting prices up beyond even what the middle class can afford. Building new houses won't fix this as the new houses will end up being owned by the rich as well.

If we define 'healthy market' as one in which there are abundant buyers and sellers, and in which prices on the lower end are consistently within the middle class's grasp, is inequality still bad in and of itself?

Yes. As an analogy, inequality has been linked to lower average adult height. Silent, small deficits in nutrition and health lead to lower height attainment among most of the population; the wealthy tend to be quite tall, but there's only so much environmental optimization can push even people with genes that predispose them to a high final height.

In housing, you have many people struggling to afford to buy anything at all, while investors buy up multiple properties or drive prices for even modest residences into the 7-figure range in some areas.

Essentially, unequal societies bifurcate, and while the vast majority of people have to go without the resources they need to lead fulfilled and dignified lives, a small minority of elites put more and more resources towards trying to eke out smaller and smaller advantages against each other.

Irrelevant, the problem is runaway inequality, not general inequality.