News for hackers. Stuff that matters.
Re:Data Mesh Architecture by i_like_waiting on Saturday March 19, 2022 @06:24PM (#30738095)
So if I understand this correctly, data mesh is just data mart, that doesn't bring data in database as a table, but uses S3 storage instead (I assume because thats cheaper in the cloud?)
Re:Data Mesh Architecture by skrrr on Saturday March 19, 2022 @06:54PM (#30738301)
That + a central data platform team that provides infra, quality monitors, data lineage and catalogue capabilities + a central team that provides guidelines on SLAs, metadata standards etc. Sounds good in theory, I am eager to see how it fails in practice
Re:Data Mesh Architecture by KmVFIz on Saturday March 19, 2022 @08:58PM (#30738995)
I can chime in as part of the central team for SLAs, etc. We offer a platform to produce datasets given some inputs, SQL, and pushes to downstream systems. Standardized jobs are ran after the user’s SQL to produce standardized outputs.

It works well, but has many issues too. User’s SQLs and input data can differ, often in unpredictable ways, because they bring their own and expect the central team to handle the rest. Those edge cases break the standardization rules, fails the workflow, confuses the user because the platform is a black box, and they ask about changing it or adding a new feature. Now your standardization asks are bottle-necked by this central team, and the options are:

- to wait for the central team to fix/improve it

- find some hack around the platform

- don’t use the platform and its associated toolings, so you build it yourself and have another disjoint system for a specific use case

- central team might build a feature that one team asked for 1 years ago, but now nobody needs it anymore and nobody knows why it’s in code. Repeat many many times for various asks over the years and your code base is likely a foreign mess.

- give your resources/funding to the central team to prioritize your ask. When built and a few years later, the central team owns something they themselves never wanted.

Re:Data Mesh Architecture by shoo on Saturday March 19, 2022 @08:25PM (#30738817)
many points of internet karma (and perhaps a profitable career as a consultant) awaits anyone who spills the beans on how their grand data mesh rearchitecture actually turned out a few years down the track, and if the exciting new problems caused by the data mesh were easier or harder to deal with than the boring old problems caused by the organisational and IT architecture it replaced.
Re:Data Mesh Architecture by Foobar8568 on Sunday March 20, 2022 @02:20AM (#30740580)
You mean like the shit show that is Data Vault? https://danlinstedt.com/allposts/datavaultcat/datavault-issu...

>>data vault 2.0 brings with it methodology, architecture, modeling, and implementation – best practices, standards, automation and more. the ability to encompass and leverage disciplined agile delivery, and sei/cmmi, six sigma, lean initiatives, cycle time reduction, and proper build practices lead us to one day sprint cycles.

And let's not forget that shit show of a "book" https://www.amazon.com/Data-Architecture-Primer-Scientist-Wa...