This is why open windows are represented on the second level, e.g. listed in the context menu that appears when you right click a dock icon. Programs and windows are not synonymous, and in fact windows belong to programs.
Macs have used this model since they first gained multitasking decades ago.
That's simply not the case.
The dock adds a dot to the application icon if the application is running.
If you have multiple instance of the application running, there will be multiple icons (each with a dot) in the dock. This is fairly difficult to do, the only way I know how is by starting the application again from the command line. Well, or when you're running an application you are developing from Xcode.
So not 100% sure what you mean with "multiple instances", but almost certainly not actual multiple instances. I am guessing you mean an application with multiple documents open.
So basically, what you regard as "bad UX" is simply you trying to map assumptions from one environment onto a different environment, and finding that your assumptions don't reflect reality.
And yes, the application will be running (showing a dot) whenever you have 1 or more documents open in the app.
To switch between documents of an application, you can either go to the application's "Window" menu and chose one from there, or to the dock icon and choose one from there.
I have many examples, but I'll just pick one. When having multiple instances of an app, the doc adds a "dot" by the app icon. This dot is present no matter whether I have 2 instances or 10 instances. Oftentimes I forget instances there (which can open due to many reasons, including for instance private browsing) because it's not easy to tell that I have 3, instead of my usual 2 instances.
Additionally, switching instances is unnecessary slow and requires more clicks than the older Windows taskbar (which is hopefully coming back as MSFT imitated Apple's terrible UX). Maybe you love this but I objectively lose productivity with this UX decision.