When I look at what you've written under those restrictions, the biggest problem I flag as a general software engineer is how the events get created, defined, listened to, etc. This tells me that if I was building something "large" like that, the first third-party library I'd want to integrate is something for handling those event streams in a more rigorous and robust fashion.
In my view, the size or complexity of the app has nothing to do with whether or not you should use a javascript/css framework. I believe it's more about having the experience & leadership required to pull the team together on complex, open-web technologies relative to the problem you are trying to solve.
Once a pattern has been established, its exceedingly easy for the junior developer to catch on and help out. Starting from zero is where most seem to struggle with the web. I've never had someone come to me and complain about the difficulty around adding a new field to an existing web form.
We were using a blend of Angular JS, Riot JS and Blazor over a period of ~8 years before we had the experience & confidence to dive into a 100% vanilla JS product. I can't fault anyone for reaching for something off the shelf to get started. But make no mistake - If you have the team, opportunity & experience to pull it off, vanilla JS products are the most enjoyable to develop & maintain. It is incredibly empowering relative to managing something like an Angular 1.x/2.x codebase, or even a modern Blazor server-side app.
I will also say this - The state of the open web standards is amazing in 2022, so you might be able to shortcut the training wheels a whole lot faster. Imagine if you had CSS grid from the very beginning. There are a lot of self-professed web developers who don't even know about this today because of the layers of ridiculous abstractions they are buried underneath.
whatever method you choose to solve this in a uniform, non ad-hoc manner will be your invented version of existing vdom or dom template frameworks, which save you from this madness. i use direct DOM only for writing high per libraries and when there is a need to optimize beyond what is provided by the framework.
i would like to see an open source, large web app written by a dozen+ engineers that manages without any ui layer abstraction and only uses vanillajs + discipline (jQuery+structure style!); in my experience this "utopia" is mostly a pipe dream, and rarely well justified.
bare DOM works fine up to a certain size/complexity app, and up to a certain size dev team in the single digits.