Ask HN: What does your BI stack look like?

Re:Ask HN: What does your BI stack look like? by sixo on Tuesday November 12, 2019 @11:03AM (#21515116)

Custom pipes mostly coordinated by Luigi (heavily customized) -> Redshift -> DBT -> Looker.

Some spark mixed in at the ingestion and transformation phases.

Like someone else said in this thread, we're currently battling Redshift scaling problems and are looking to offload more of the raw data to S3 using Spark to create read views.

No data catalog right now but the Looker setup is extremely well-maintained. Hoping to deploy Lyft's Amundsen to fix this more generally.

[ Parent | on HN ]

Re:Ask HN: What does your BI stack look like? by buremba on Tuesday November 12, 2019 @11:09AM (#21515174)

Since your Looker setup is well-maintained, did you consider adopting the data catalog in Looker, what would be the challenge doing it?

[ Parent | on HN ]

Re:Ask HN: What does your BI stack look like? by sixo on Tuesday November 12, 2019 @11:34AM (#21515406)

The main reason is that only a subset of all our datasets are actually exposed in Looker, and Amundsen would have applicability to other data sources - certain production semi-OLAP DBs, the future hive-on-S3 deployment... that we have no plans to expose in Looker at all.

That said I haven't looked closely at how far we could go with Looker, I'll check it out.

[ Parent | on HN ]

Re:Ask HN: What does your BI stack look like? by buremba on Tuesday November 12, 2019 @01:48PM (#21516842)

I see. We actually develop a BI tool similar to Looker but also want to implement a data catalog feature embedded into it. I'm familiar with Amundsen but I believe that this kind of tool needs to have tight integration with the BI tool and Amundsen still has a way in terms of integrations. (searching among saved reports, dashboards and collaboration with teams, etc.). Otherwise, using spreadsheets is still easier and maintainable.

I would love to hear the features that you would need in a data catalog system.

[ Parent | on HN ]

Re:Ask HN: What does your BI stack look like? by mgrover on Tuesday November 12, 2019 @05:01PM (#21518641)

One of the Amundsen people here - happy to answer questions or hear feedback, if there's anything.

Agreed there's more to be done in terms of integrations. Our integrations are pretty focussed on databases and people (https://github.com/lyft/amundsen#supported-connectors). Work on indexing dashboards is being scoped, you can see a rough product spec with a rough mock here: https://docs.google.com/document/d/16cSKgM2sCYvhKq54yfwaHKws...

Feedback welcome!

[ Parent | on HN ]