News for hackers. Stuff that matters.
Re:Is BI dead? – On dismantling data's ship of Theseus by marcinzm on Tuesday September 21, 2021 @03:11PM (#28608671)
I run a data team. We do two things:

* Provide a clean minimal view of the data that other teams can pivot table on top of or download in the BI tool. We do onboarding, support and all that for this data.

* Provide dashboards and reports for more complicated ongoing questions that require data that's not cleaned yet.

We don't use Looker but based on the sales demo it seems optimized for this type of workflow. The core analytics team maintains the data and complex workflows while other teams have business analysts for day to day questions.

Re:Is BI dead? – On dismantling data's ship of Theseus by haddr on Tuesday September 21, 2021 @03:20PM (#28608759)
This is the direction where most BI work is heading for: you need a team of data engineers that can make sure that the data for your BI dashboards is delivered and in a good quality. Everything else that doesn’t need it is probably simple enough to be run as some PowerBI dashboard on top of production DB. I think today most effort in those heavier BI cases is spent on data transformations and making sure that data flows/batch jobs run uninterrupted.
Re:Is BI dead? – On dismantling data's ship of Theseus by dikei on Tuesday September 21, 2021 @10:38PM (#28612568)
If you're not careful, the data engineers team is often overloaded with requests from the analysis team.

With big data warehouse, more often than not, the fancy dashboards would not run fast enough without a dedicate pipeline that materialize data before hand. Too many useless dashboards and your data team would not have enough manpower to do anything else other than operating the existing pipelines. As a result, new dashboards become slower than old dashboards due to lack of maintenance/optimization, and everyone complain about the poor performance.

Re:Is BI dead? – On dismantling data's ship of Theseus by dspillett on Wednesday September 22, 2021 @06:27AM (#28614787)
> dashboard on top of production DB

A lot of what users need can be done this way in my experience. Also giving their people access to customise reports/exports can go even further. I'd advise against direct access to “real” production though: have the dashboard and custom reports run against a (probably readonly) replica of some sort¹ so runaway reports don't impact live app performance.

[1] via replication, log shipping, etc... — all major databases should support at least one method that can achieve this with minimal latency in updating the replica under normal conditions

Re:Is BI dead? – On dismantling data's ship of Theseus by marcinzm on Wednesday September 22, 2021 @08:36AM (#28615591)
>A lot of what users need can be done this way in my experience.

In my experience, no, as the company gets to any complexity and size. The production DB is often optimized for use by the ORM/language of choice or there's twenty of them for microservices. This makes it really awkward to do some things in SQL. So now your engineering team is getting requests to keep changing the prod DB schema so analytics can be done more easily. More over the prod DB is likely 10% of the data your business needs. There's a reason companies like Fivetran exist to slurp fifty data sources into a warehouse. Marketing wants google and facebook and mailchimp data. Operations wants saleforce data. Finance wants stripe data. And they want all of this tied to the inhouse data.

Sure you can make reports in each of those services but then you need to tie them together and then people complain of differences in metrics and then you have a bad time.

edit: Not to mention analytics on what users do. Always fun when there's a billion row table in your production db for that.

Re:Is BI dead? – On dismantling data's ship of Theseus by Guest42 on Tuesday September 21, 2021 @04:14PM (#28609384)
Having used looker, it felt rather clunky and proprietary

I was generally able to create better results via power bi, ssrs, or shiny

Re:Is BI dead? – On dismantling data's ship of Theseus by marcinzm on Tuesday September 21, 2021 @04:47PM (#28609714)
As I understood it the goal of Looker isn't to make the lives of people who can use something like Shiny better but rather the lives of people who use pivot tables 99% of the time. So rather than having technical analysts making dashboards you have business analysts making dashboards (or at least digging into generic dashboards).