When it comes to business stakeholders, the biggest obstacle is trust. If you're not a data person, or a developer, decision logic being gated behind code is a scary black box. "If I can't control it, I can't trust it". That feeling gets even worse when we build systems that are controlled by AI or ML.
I think we need more solutions for data teams that allow business stakeholders to take part in the automated workflow deployment. Let them see how things are connected together. Let them verify that the decisions are being made correctly every day. Let them tweak levers so they have a say in how things are working. That's the only way to move beyond the current environment where everyone wants a dashboard, but no one looks at it.
That's a big problem I'm aiming to solve right now.
The technology is getting good enough with cloud warehouses that most of the logic can be defined in SQL, with a script only needing to map the results back to an API.
Current domain is SaaS. I've found many of the same types of opportunities on the sales/marketing side of things, but not at quite the same scale.
Agreed that automation it will always take more time than a visualization. If we can shift the conversation towards the expected action that will be taken off the dashboard, we can hopefully save thousands of wasted hours on visualizations that don't get used, redirecting those hours towards more impactful efforts.
I would love to expand my mindset and hear of domains where the complexity is far higher. Feel free to shoot me an email.
Essentially, I first implement the engine with the output being an automated list of 1-click actions (typically a set of links on an email, but could easily be a dashboard button or anything else), so the system default is "detect, but don't do". After an evaluation period of the system actions, we move onto the second part, in which the system performs the actions and produces a log of activity, plus a list of 1-click UNDO actions to the user.
The idea is that a) the system earns trust from the stakeholders due to their direct involvement, b) the system can enjoy some supervised learning from someone other than the dev team and c) worst case, if it never earns enough trust, I've still saved a stakeholder dozens of hours of work sifting through dashboards as opposed to taking a look at a pre-filtered list. A few systems have turned out well enough that they never move from Phase1 while still being considered a massive win.
Is this all internally built?
I think the visualisation matters though. It's much easier to convince people that the automated actions are worth doing once the related data is visualised. And at that point, whatever provides you both may be the best solution.
Is the platform still easily used by business users? Or does it primarily become a gated way for Data Engineers to build/action on data?
A lot of decisions can't be easily visualized in your standard dashboard. You can forecast the potential of those automated decisions, but that's still black box logic. You really need more of a middle ground that lets you preview "given these inputs, show me what the output would be".
I don't know and honestly I don't believe there's a system that actually achieves it. But this it does go beyond BI in actually being aimed towards running code based on results. Which makes it closer to Labview-with-graphs than to BI.
I reckon of the 90% that don't make any data driven decisions with the dashboards they're provided, there are three causes I run into frequently.
1. They can't describe what info they need
2. That info isn't available.
3. They don't make decisions based on data at all.
I feel I'm getting better at addressing 1 and 2 the more I do it, but I figure there's a fair amount from group 3 who demand the reports nonetheless...
As an aside, you mention Tableau, QlikView or PowerBI, have you tried Apache Superset at all?
It’s a very important function: provide data that can be cherry-picked to support a decision already made.
If the decision is too complex for that, hire a big consulting firm to write a report suggesting and justifying the action desired.
A decison staple surely going back millennia.
One way to solve is to automate the action from insight part and inject into users workflow. I am looking at this now with support of senior sales stakeholders.
Simple example:
Client X activity has been trending down vs it's peer group, Call Client X to understand what's going on.
Nobody is saying "I'd love another tool that I am expected to check every day", both embedded analytics and insights are how to drive adoption and unlock value from data.
This is part of the expectation of business analysts, that they will surface and communicate data not immediately being asked for by executives or other stakeholders. This usually means running a meeting and presenting findings, and then pairing with another stakeholder to flesh out further opportunity.
If there's no one to pick up the baton, there's very little that can be done about that.
It feels like a good idea, but I imagine it would be very fragile in black swan events.
Maybe a mix?
Provide the dashboards, but allow users to set up alerts/workflows, they understand the problem well, and if they know what they're going to be going to the dash to find, let them skip that step. In case something new happens, they can set up a new alert/workflow/whatever for the next time it happens.
Only works if you've got buy-in from management though.
So then we have to get our db people in to rewrite all of their reports so they don't do this which eliminated the benefit of having the customers do it.
In the end there is just no avoiding having experts build the dashboards.
Interesting as we're looking at going long with PowerBI embedded for a client facing app and ditching our home grown effort which is always playing catch-up and costs a fortune.
* Provide a clean minimal view of the data that other teams can pivot table on top of or download in the BI tool. We do onboarding, support and all that for this data.
* Provide dashboards and reports for more complicated ongoing questions that require data that's not cleaned yet.
We don't use Looker but based on the sales demo it seems optimized for this type of workflow. The core analytics team maintains the data and complex workflows while other teams have business analysts for day to day questions.
With big data warehouse, more often than not, the fancy dashboards would not run fast enough without a dedicate pipeline that materialize data before hand. Too many useless dashboards and your data team would not have enough manpower to do anything else other than operating the existing pipelines. As a result, new dashboards become slower than old dashboards due to lack of maintenance/optimization, and everyone complain about the poor performance.
A lot of what users need can be done this way in my experience. Also giving their people access to customise reports/exports can go even further. I'd advise against direct access to “real” production though: have the dashboard and custom reports run against a (probably readonly) replica of some sort¹ so runaway reports don't impact live app performance.
[1] via replication, log shipping, etc... — all major databases should support at least one method that can achieve this with minimal latency in updating the replica under normal conditions
In my experience, no, as the company gets to any complexity and size. The production DB is often optimized for use by the ORM/language of choice or there's twenty of them for microservices. This makes it really awkward to do some things in SQL. So now your engineering team is getting requests to keep changing the prod DB schema so analytics can be done more easily. More over the prod DB is likely 10% of the data your business needs. There's a reason companies like Fivetran exist to slurp fifty data sources into a warehouse. Marketing wants google and facebook and mailchimp data. Operations wants saleforce data. Finance wants stripe data. And they want all of this tied to the inhouse data.
Sure you can make reports in each of those services but then you need to tie them together and then people complain of differences in metrics and then you have a bad time.
edit: Not to mention analytics on what users do. Always fun when there's a billion row table in your production db for that.
I was generally able to create better results via power bi, ssrs, or shiny
Yes. A lot of darts are being thrown as to where XR fits on which Hypecycles, but one of the most promising uses to me is as a new enduser engagement layer for BI(g) data. The new UI/UX possibilities of immersive 3D have a low level effect on what/how information can be rendered in meaningful ways. It's still early days in this space, but here is a quick list of some work in progress for anyone interested:
[1] https://flowimmersive.com/ Focused on Data driven story-telling through an inhouse 3D visualization tool. Strong on AR collaboration and responsible for "The Data Guy" popular on TikTok.
[2] https://www.badvr.com/ Industry focused on using XR for specific applications like 5G Radio Coverage heatmap modeling and Smart City interfacing tools.
[3] https://3data.io/ Use of XR for IT Operations Center applications (NOC/SOC)
[4] https://github.com/ACEMS/r2vr This project lets you output basic WebXR 3D visualizations using R
But most people seem to log in once or twice and then never again. And if you ask why it’s often because they couldn’t find the answer to their incredibly specific one-time only question, so they decided it’s all useless and they continue to work on their own personal homegrown BI-suite in excel.
Also, we use Qlik and I absolutely hate it. Luckily I don’t have to work with it too often but whenever I do I feel like I’m always fighting it. Using Power BI and Tableau felt much more like working together to solve issues.
Hah this is probably me. The university I work for uses a BI tool (MicroStrategy) to track students/majors/etc. But I usually find it easier to use the BI tool primarily as a way to get a CSV or XLS export of enough source data to answer my question, and then do the actual analysis in Python+pandas. I can probably formulate my query in their weird proprietary query builder, but it seems unnecessarily complicated and I already know how to do data analysis in Python.
If you're talking about Power Query, the best way to use it is to write code directly once you observe what the interface generates. It's a rather nice kind-of-functional language that really comes into its own when you start using/creating higher order functions.
Power Query gave me an intense craving for a form of SQL with a similar facility for functions.
For example, I have a lot of experience writing Oracle SQL, and Power Query didn't offer "grouping sets". But I realized it could be implemented using functions. It would be so great if SQL supported that.
Why aren't you providing your dashboards via Excel, then? What exactly does the fancy BI software do that Excel won't? I have used Qlikview but not Tableau or Informatica, and I'm a bit fuzzy on what people mean when they say "Power BI". Is using Power Query what you would call Power BI? Or does that mean using the data model stuff? Power Query is the part of Excel that I particularly like because data kind of stays where you put it.
Power Query is the only part of this that has seemed useful to me.
The Data Model and DAX in particular seemed to me like distractions that I wasted time on before discovering the M language.
Saying Power Query is mostly unrelated demonstrates that the confusion is not all mine.
I've used both DAX and M (Power Query) via recent versions of Excel. Obviously there are plenty of ways to interface with data.
https://en.wikipedia.org/wiki/Power_Pivot says both Power Query and Power Pivot are part of Excel.
What I'm seeing there looks as though it is targeted at people who make the decision to buy products like this, but not to use them. I'm not seeing what couldn't be done with Excel 365 alone. There are screenshots with verbiage about how well it works with Excel. And they look like...Excel.
I guess it supports R, and maybe makes it easier to seamlessly refresh a report. That's aggravating though, that it makes me suspect Excel might be purposely rough around the edges in order to not displace Power BI.
As I've said a few times, I like M and wish they'd commit to it.
I honestly suspect Microsoft is repackaging functionality that already existed for many years in their "ball of mud" software, with a shiny new wrapper and better defaults, but practically unusable, because people who license software are easily manipulated.
Power Automate is the most awful example of this I've used so far.
You say it's a competitor to Tableau, well, Tableau is a competitor to Qlikview, and I've used Qlikview and Excel enough to consider Excel better in every way that I care about. (My organization uses SharePoint, so that's how reports are usually distributed)
Allowing stakeholders some leeway to conduct their own independent analysis (after a short training session) has allowed our strained data team to hand over simpler analysis to focus on the harder problems.
Less is more.
Firstly Tableau, QlikView or PowerBI are all pretty much doing the same thing whatever flavour you prefer.
We find that maybe 10% of users will actually use them to garner new actionable insights and 90% will moan there's too much data for them to process and no one appreciates the immense amount of data munging that goes on behind the scenes to make those pretty graphs.
Where do we go? Personally I see a lot of the insights being automated and turned into actions on the server side without anyone touching a BI tool. This can be achieved with rules based approach and perhaps some correlation, trend analysis over time series data. Where you do need to go beyond that then perhaps AR/VR will provide a novel and more valuable approach.