Re:Show HN: Metabase, an open-source business intelligence tool
by jtmarmon on Wednesday October 21, 2015 @03:17PM
(#10427864)
got it. so the idea is generally people want to do sort-of 2nd derivative queries on the data, so it's best to get those first stats out of the way in the warehousing step
Re:Show HN: Metabase, an open-source business intelligence tool
by vog on Sunday October 25, 2015 @03:25AM
(#10446260)
Yes, although I wouldn't describe this as "2nd derivative queries", but more like "put the code that you need anyway into two separate layers (tools) with clean boundaries and a persistent intermediate result".
The data warehouse contains datasets in a unified (and usually simplified and reduced) form, defined by the needs of analysis tools.
If your analysis tool accesses the data lake directly, it will almost certainly contain "parsers" for various operational data formats. Also, it will perform those transformations over and over again every time. And multiple analysis scripts may contain multiple versions of those parsers. The idea is to separate these "parsers" out of the analysis step and to "cache" the cleaned-up intermediate result. That "cache of clean data" is usually called "data warehouse", and can create good indexes on that data, multiple runs of your analysis tools have very fast access.