News for hackers. Stuff that matters.
Re:The Rise and Fall of the OLAP Cube (2020) by mwexler on Monday July 05, 2021 @01:40PM (#27740150)
"We model data according to rigorous frameworks like Kimball or Inmon because we must regularly construct OLAP cubes for our analyses."

Actually, no. We model our data this way so it can be used for business decisions. It doesn't take long for any entity of any scale to discover that the logging and eventing done for heartbeat status, debugging, and scaling is just different than what you need to make a variety of business decisions.

You can solve with more events at different scales (button clicks nested in screen views) or pick events or event rolkups that appear to be clean business stages ("completed checkout") but still, your finance team, marketing group, all have different needs.

So, you decide to have some core shared metrics, derived and defined, and make them usable by everyone. Folks agree on the defns, and due to ease and trust, you see more data supporting more decisions.

You discover that some folks are doing 10 table joins to get an answer; it's fast but difficult to extend for new questions. You decide to build a view that solves some of these pains, and refactoring to allow a better time dimension. Your version links with the metrics you created, and the resulting queries shed tons of CTEs while becoming readable to the average user.

And now, you have some ELT pipelines, some event transforms that result in counts and filters that map nicely to your business needs but still allow you to get atomic raws, and you and your teams start to trust in consistent results. Your metrics are mostly clearly summable, and ones that aren't are in table views that precalc the "daily uniques" or other metrics that may need a bit special handling.

You've started modeling your data.

No, we don't need olap cubes. But we do need some type of rigor around analytic data. Otherwise, why go to all the trouble to collect it, count it, and predict from it if it may be wrong with no measure of that uncertainty?

And yeah, Kimball et al are from a world where olap was the answr, but it turns out they solved a broader set of sql problems. So, worth learning the good, toss the dated, and see what good data modeling can do for your analysis and predictions.

Re:The Rise and Fall of the OLAP Cube (2020) by glogla on Monday July 05, 2021 @06:45PM (#27742538)
> Actually, no. We model our data this way so it can be used for business decisions

Yeah, that would be my disagreement as well.

Sure, some of Kimball might be obsoleted by modern technology. But I don't think the point of Kimball is cubes, and even if it was, what better is there?

I'd be really interested in what else is there, that is more modern and suited to the modern world.

Data Vault? That probably isn't it, most of all because it's all about the "platform to build you Kimball on" but not the actual business modelling. (But the parts about Metrics Vault and Quality Vault and the "service stuff" are really good.)

Anchor Modelling? That one seems, looking by 2021 eyes, like a way to implement columnar storage and schema-on-read in MS SQL Server using store procedures ... which is probably not actually a good idea.

Puppini Bridge aka. Unified Star Schema seems like interesting concept, especially in self-service space. But even it's proponents warn you it doesn't scale performance-wise and also it is kind of incremental change on Kimball. (But the way the bridge table is kinda adjacency matrix of graph of your objects tickles my inner computer scientist fancy)

So really, what else is there?

Re:The Rise and Fall of the OLAP Cube (2020) by cedricd on Monday July 05, 2021 @08:38PM (#27743186)
This isn't widely known, but activity schema[1] is aiming for this space (disclosure - I co-founded the company supporting it).

It's a modeling approach that separates data modeling from data querying -- meaning that once data is modeled, it can answer any number of questions. The base building blocks of the data model can be combined at query time without having to figure out explicit foreign key joins.

[1] https://www.narrator.ai/activity-schema/

Re:The Rise and Fall of the OLAP Cube (2020) by impjohn on Tuesday July 06, 2021 @06:24AM (#27746382)
Have you looked into the VKG (Virtual Knowledge Graph) approach?
Re:The Rise and Fall of the OLAP Cube (2020) by mwexler on Monday July 05, 2021 @01:50PM (#27740248)
BTW, their followup piece at https://www.holistics.io/blog/olap-is-not-olap-cube/ is a nicer drill down into some of these issues. Also a good read.
Re:The Rise and Fall of the OLAP Cube (2020) by tresil on Monday July 05, 2021 @03:53PM (#27741289)
Could not have written this any better. I’ve been in the data business for over 10 years. A number of technologies have come and gone during that time, but dimensional modeling and solid SQL skills keep delivering remarkable value.
Re:The Rise and Fall of the OLAP Cube (2020) by chrstr on Monday July 05, 2021 @05:23PM (#27741949)
Indeed, it is generally good for analytics to have a conceptual data model that is reasonably efficient both on the technical side (to quickly get answers for queries), and also on the logical side (to quickly translate questions to queries).

This model doesn't need to be OLAP cubes, but it's also not that easy to find something better.

Re:The Rise and Fall of the OLAP Cube (2020) by hodgesrm on Wednesday July 07, 2021 @10:21AM (#27761434)
> No, we don't need olap cubes. But we do need some type of rigor around analytic data. Otherwise, why go to all the trouble to collect it, count it, and predict from it if it may be wrong with no measure of that uncertainty?

Modern data warehouses don't need to build cubes for performance reasons. Low latency data warehouses like ClickHouse or Druid can aggregate directly off source data. The biggest driver for modeling is allowing non-coders to access data and perform their own analyses. I don't see that problem ever going away. Cube modeling with dimensions and measures solves it well.