What the Heck is a Data Mesh?

Re:What the Heck is a Data Mesh? by zwkrt on Thursday July 29, 2021 @04:06PM (#28000822)

In my experience at three large companies, any project where one part of the organization wants “the data” from another is actually just a power grab at the mid-manager level. To me when I hear “accounting wants direct access to the inventory data” I interpret that as cuz “accounting manager thinks the inventory team is slow or incompetent and thinks if their own team just had the underlying inventory data directly accessible she could cut out the middle man!”

The problem of course is that data has to be interpreted, and often that interpretation is complicated. After all, that is why we write programs and don’t just query/insert into databases directly from the terminal. Most “data” is inextricably tied to the programs that interact with them, and freeing the data without making the complexities of the program known leaves both organizations open to horrible bugs.

[ Parent | on HN ]

Re:What the Heck is a Data Mesh? by na85 on Friday July 30, 2021 @01:42AM (#28004831)

I've been on both sides of this coin. As the requestor it's usually because I'm fucking tired of bureaucratic controls and shitty Oracle databases from 1982 running on Windows XP that won't let me execute arbitrary queries "because security".

I just want to do my job more efficiently instead of transcribing aircraft serviceability data by hand into excel.

[ Parent | on HN ]

Re:What the Heck is a Data Mesh? by datameshlearn on Thursday July 29, 2021 @07:28PM (#28002781)

That's kind of one of the reasons for data mesh. The domain is the one who controls how the data is stored and made available so they get to show off how useful their data is and might get some great insights back. But if the team is so lacking in empathy, that data mesh implementation will almost certainly fail. So there needs to be at least some buy-in but if you can convince a domain that participating is a public good (which many do) and that they actually have more control like this (they get data engineers added or at least embedded in their team), it can be gravy/groovy.

[ Parent | on HN ]

Re:What the Heck is a Data Mesh? by tomrod on Thursday July 29, 2021 @09:47PM (#28003575)

Seems like data mesh assumes a culture of good will and acting in good faith,

[ Parent | on HN ]

Re:What the Heck is a Data Mesh? by alexisread on Thursday July 29, 2021 @07:26PM (#28002759)

This makes a good case for visible data lineage (external system coupling), in conjunction with clear program/ETL documentation (internal data coupling), so you can see the full data transformation.

There are a few cross cutting concerns with a data mesh, namely authz, schema and cacheing. Most companies don't consider the data mesh at a company level which is a shame as solving all the above should be doable at a company level.

[ Parent | on HN ]