1. Implement a sane (albeit less powerful) view hierarchy system, foregoing basically all of the CSS selector stuff, and applying styles individually. The framework will be big and unwieldy, and you'll take a performance hit, but you'll be able to maintain the client code.
2. Bite the bullet and implement it the ad hoc way, comment the crap out of it, and hope that whoever comes and changes something later will remember to read those comments.
Imperatively generating and dealing with a DOM structure using simple selector queries is relatively easy to do, as is modeling what is happening in your head. Dealing with more complex CSS rules as they apply to a static (or dynamic but very simple) DOM structure is a little trickier, but not completely unwieldy. When you run into problems is when you try to do both simultaneously. Suddenly you have to not only mentally model a mutable DOM structure, but you have to model how those CSS rules will be applied in a generalized and abstract form. It's not hard to paint yourself into a corner where you'd actually start having to prove theorems about your code in order to be reasonably assured that the CSS will apply correctly.
Of course, hardly anybody actually lets it get that far. Instead, they start applying styles imperatively using JavaScript. And that sucks.
I don't know what the solution to this is, or if there even can be one that is a progression from where we currently are. But I don't think fancier CSS selectors are the answer.
Like you, i can't think of a good solution - a lot of UI frameworks either just has really heavy restrictions, or it doesn't actually provide enough abstraction to actually help with the problem. Sigh...