Soft costs and impact fees for a home in San Francisco or New York can easily be in that range.

Sure, but a permit and inspection does not cost 500K. Making a lot conform might cost that much, sure.

Poster here, 500k was not intended as literal or typical, just and example of how regulatory costs can add to market price.

That said, I have spoken to builders who described city costs of several hundred thousand dollars for renovating or building single family/duplex homes. For start, SF permit costs are a % of construction with 6-9% for the basic permit. If you do the work yourself, you have to pay a % of the market cost if you were to contract it out. Then there are a plethora of additional fees and permits, and you can find some of them here {1}.

I think you are right that the single biggest cost is code conformance, minimum size, construction ect. Some of the requirements are ridiculous. For example California requires electrical outlets every 48 inches along countertops, independent of if I want them at all. This mentality extends to every part of the the permitting process.

https://sf.gov/step-by-step/get-building-permit-house-review