The original OLTP/OLAP dichotomy was based on architectural tradeoffs required for spinning disk and the way indexes worked. Especially at the time Vertica was created, the OLTP-ness of PostgreSQL was essentially hardcoded into the architecture. Vertica made a lot of changes, e.g. to storage behavior, to support their use case that would have significantly impacted OLTP performance. A complicated query that mixes NSM and DSM tables would be unreasonably complex since the query building blocks are different depending on the type of storage.

This trade off doesn't really need to exist in a SQL database today on modern hardware with its extremely high storage bandwidth. There are other models that can satisfy both OLTP and OLAP use cases satisfactorily in a single coherent system with modern internals. The SQL database market is extremely conservative, so even if you built it no one would adopt it for a decade.