News for hackers. Stuff that matters.
Re:Measuring the popularity of different vector databases by cmrdporcupine on Tuesday September 20, 2022 @03:03PM (#32916344)
Ok all that said, this is what I've heard before, but what is the actual technical limitation behind "simply" providing alternate columnar storage&index implementations for tables optimized for OLAP but sharing the surrounding query parser, query execution framework, and potentially even query planner that is used elsewhere for OLTP?

Like... Vertica is a fork of Postgres. I'm curious why they chose to fork and implement column-oriented storage and indexes etc. rather than simply add them as options to stock Postgres.

Obviously joins across the two different worlds would be highly problematic, and perhaps query execution, data materialization, query planning etc. could look significantly different.

But the potential advantage to the end user seems high, if moving from "transactional" to "analytical" workloads is a matter of moving data from one table-type to another, within the same underlying database system.

Again, I know there are reasons why this approach has not been successful. I'm curious what they are.

Re:Measuring the popularity of different vector databases by jandrewrogers on Tuesday September 20, 2022 @04:05PM (#32917062)
The original OLTP/OLAP dichotomy was based on architectural tradeoffs required for spinning disk and the way indexes worked. Especially at the time Vertica was created, the OLTP-ness of PostgreSQL was essentially hardcoded into the architecture. Vertica made a lot of changes, e.g. to storage behavior, to support their use case that would have significantly impacted OLTP performance. A complicated query that mixes NSM and DSM tables would be unreasonably complex since the query building blocks are different depending on the type of storage.

This trade off doesn't really need to exist in a SQL database today on modern hardware with its extremely high storage bandwidth. There are other models that can satisfy both OLTP and OLAP use cases satisfactorily in a single coherent system with modern internals. The SQL database market is extremely conservative, so even if you built it no one would adopt it for a decade.

Re:Measuring the popularity of different vector databases by heisjustsosmart on Thursday September 22, 2022 @05:37PM (#32944947)
"There are other models that can satisfy both OLTP and OLAP use cases satisfactorily in a single coherent system with modern internals"

ok, which are these.

Re:Measuring the popularity of different vector databases by convolvatron on Tuesday September 20, 2022 @03:47PM (#32916842)
most of it i think is just focus. its alot of work to pull together an OLAP and an OLTP database and they are very different workloads.

the one place where you do start to get into fundamental issues is consistency. the planner can certainly identify read-only transactions and statically remove some conflicts. but whatever scheme you are using (locks, mvcc, optimistic) is going to struggle the more concurrent overlapping transactions there are. since OLAP transactions are very long lived and touch alot of things - they are pretty hostile co-residents with the OLTP traffic.