How to put it? I know where the state really lives in the real DOM. I have HTMLInputElement and if user edits it, I can always access HTMLInputElement.value and have the current value. I can associate my data directly with the DOM nodes as simply as node.myData = {something}; and it stays there no matter where the DOM node moves in the meantime.

Virtual DOM is easy to reason about in this simple case:

Events -> State -> VDOM -> DOM

But this is an illusion. DOM nodes can change state as a result of user interaction and you need to account for that (in your head and by updating the State to match).

Anything to do with DnD and animation at once seems like pain to think about in the VDOM world. Or any visual manipulation of dom nodes, reordering, nesting, grouping, etc. DOM has useful preexisting interface for moving nodes around the tree. Easy to use and directly related to what you see on the screen.

There are many cases where abstracting the state away from real DOM and pushing it down in the one way manner of VDOM just doesn't bring any benefits.

In the real DOM world you can glue the state/data to the visual elements on the page, write code that creates UI for re-arranging the visual DOM nodes and then at any time collect the data back from the DOM nodes in the visual or any appropriate order with powerful DOM traversal methods. This direct two way association of DOM and my JS objects helps in so many cases.

Sometimes it is easier to solve the problem in the visual space (DOM nodes) and sometimes it is easier to go from my custom JS objects representing a complex component or something. With VDOM this flexibility is lost and as a result some problems seem like requiring me to stuff the square peg into a round hole. I guess it is somehow possible, but what's the benefit?

You seem to be saying that VDOM gives no benefits, but in my experience the idea that your view is a function of your state that you can just completely recalculate whenever your state changes is a hugely useful abstraction that makes code much less bug-prone. And it is not available without VDOM because the performance would be terrible if you generated actual DOM this way and you would also lose state like scroll state that might not be reflected in your store. No doubt there are some tricky cases for VDOM. Maybe some of them even warrant using it only selectively although my experience has been that when I started compromising on the purity of the React/Redux model it caused problems. (And I will grant you that this in itself is a weakness of the model because sometimes you can't control where state lives.)