If an enterprise wants to "get all the data out of the silos", unstructured storage is only "friendly" to the temptation to be sloppy in such an export.

Not only missing or corrupted individual columns, but completely incoherent and arbitrary complex document structures. If "no-one knows the schemas and no-one knows how to join between them" the data lake project has already failed to provide value.

The secret is that it doesn't matter, because many, many tech businesses aren't business but investment ponzi schemes / "moonshots" on other people's money, who are mostly arm's length idiots.

Say you have a pile of garbage in silos: user metrics, analytics, usage stats, logs, sales, revenue across three different systems, etc.

You copy the garbage out of the silos and into a lake. It's a mess. No one could tell you anything real with that data.

So you write queries that ignore the errors, and don't show what you say they show, but look nice and you say they show what you know people want to hear. You know you have sales, right? You know things are going well, right? Or they will be soon, for sure. So you're making the numbers tell the story -- any problems in the data are just bumps in the measurement.

You show investors your fabrications, they show their investors your fabrications, and so on. You get another funding round for ~1.5x one year of salary for your employees + office costs. Life goes on, you have meetings. You make presentations.

Rinse. Repeat. You sit on boards, talk about innovation, sponsor a charity. Maybe make some investments! Be a local tech celebrity. 5 years go by, your business is huge and bleeding cash, and somehow... no one's willing to invest more.

So you sell what you can for what you can get. Some investors broke even, some lost the farm, some maybe even made 3-4x. No hard feelings overall! "Our Amazing Journey" blog post. The employees move on. You move on.

Rinse, and repeat.

Garbage in, garbage out.

It doesn't matter, because you were never really running a business.

I'm sorry that you are involved with this kind of toxic "business" and with funding round beggars. I've only worked with real users of real data for real process integration, real marketing, real monitoring and auditing, under real management, in real companies.

Thanks! I appreciate that you bothered at all to respond to my many-days-later comment. :) I'm glad you've respectively had good luck in this regard.

I should be clear that I'm not speaking in particular about a present or past employer, nor do I really mean to denigrate businesses operating as I described. My rhetorical cynicism might be misread as just cynicism, so I'd like to try again, hopefully more charitably:

Not many top decision makers are (or closely employ) good stats people. So pretty optimistic non-stats folks try to figure out what to measure, how to measure it, what is meaningful, and how to extrapolate that into good forecasts -- not a great start for accurate measurement and forecasting.

Next, data is often bad: because of adblockers, brief app outages, known workarounds, bugs, gaps, measurement-method changes over time, differing cohort assumptions, etc. There always ends up being some hand waving. Another strike against accurate measurement and forecasting.

Finally, businesses must survive, and many are not cash-rich enough for safety or for growth, so in the near term they may be oriented more toward investment than revenue. Some good hand waving can then be the essential survival activity, and good hand-waving may require treating some fictions as real. The dreaded third strike falls against accurate measurement and forecasts.

So in a sense, these aren't traditional businesses. Their measurements and forecasts may not be very accurate, but for now it doesn't really matter. What matters more is usually the story.

Yeah, it's kicking the can down the road!

If you can't or won't do the work to understand and define the schema at the time of ingestion, you're asking consumers of that data to do the work for you later on, and then you're looking at an archeology project to try and figure out what the heck people were thinking in the past. It's only going to get worse as you keep shoving new data in there with its own incompatible poorly defined schema.

It probably ticks a compliance box but good luck making use of it...

I've only ever witnessed consumer-driven ad-hoc integration initiatives, where the team that needs data coerces the team that has data into providing it in a usable form, and people notice stale or incorrect feeds because they are actually in use. Why should a company invest server resources and development effort on "ingesting" data that nobody cares about in a way that doesn't fit any purpose? Attempting to collect "all" data is meaningless.

I'm horrified by the notion of banks in particular storing their data where "those datasets are not well governed, no-one knows the schemas". This is people's money you're talking about.

A lot of applications can fail with an "eh, whatcha gonna do?" shrug when records fall out because their schemas drifted or two different developers interpreted them differently. So somebody's preferences stop working or their early "likes" get forgotten; so what? Move fast, break stuff, etc.

But when your data actually matters, I'd much rather pay the expense of having some discipline about it. The idea that my money might become the stuff that gets "broken" is kinda terrifying.

All this talk of "relationships" overlooks the fact that the "relational" part of an RDMS is the mathematical concept of relations.