News for hackers. Stuff that matters.
Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by blauditore on Thursday October 08, 2020 @10:06AM (#24719187)
I get a strong buzzword bingo vibe from this post. On a related note, is there a good reason to ever have something like a "data lake" (and call it like that)? Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy name.
Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by PeterisP on Thursday October 08, 2020 @10:31AM (#24719380)
The reason for data lakes appears in large enough organizations where it becomes exceedingly likely that there is some data that may be useful to you that's maintained by people you'll never meet in a department you don't know about, where it's impractical or even impossible to get them involved in your project that would consume this data.

It's not so much about data itself as an attempt to solve a communications and coordination organizational problem; you decouple sources of data and consumers of data (not the technical systems/databases, but the people and organizational units) to a 'hub-and-spoke' model where the providers of data just supply raw data without getting into a multinational project that takes a year just to identify the potential stakeholders for that data throughout a distributed organization with tens of thousands of employees.

Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by curiousllama on Thursday October 08, 2020 @11:34AM (#24720088)
Yea, data lakes are a tech solution to an org problem. Good or bad, it’s what it is.
Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by disgruntledphd2 on Thursday October 08, 2020 @01:57PM (#24721813)
Done right, they are awesome though. As a DS you can iterate a lot faster if you don't have to access multiple different stores for features or data.
Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by slotrans on Thursday October 08, 2020 @01:51PM (#24721738)
The simple way to understand the utility of a data lake is "S3 is phenomenally cheap".

Some people treat this as an excuse to throw whatever they want into it, without any organization or standardization... and the consequences of that are quite predictable.

But it doesn't have to be that way. You can accumulate diverse and large data sets, in cheap cloud storage, while knowing what everything is and where you can find it.

As a trivial example, let's say you have a typical OLTP database (or perhaps many), with useful data that is, unfortunately, mutable. You can store entire copies of those tables in your data lake for pennies a day, giving you the ability to recall a transactionally-consistent view of that data from various past times. This is something we've always been able to do using traditional tools, the difference is that storing the data in a "data lake" (i.e. S3) is orders-of-magnitude cheaper.

Another major use case, perhaps the most significant one, is storing the raw ingested data -- e.g. from telemetry collection, 3rd party exports, etc -- along with each stage of its transformation. By preserving the original input, along with all intermediate outputs, no information is ever lost. If a buggy transformation is discovered it no longer means your output is irrevocably corrupted, fixed results can be re-computed from wherever in the transform pipeline the bug manifested. And again, this was always possible, a data lake just makes it cheap enough to actually do on a large scale.

Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by laichzeit0 on Thursday October 08, 2020 @03:47PM (#24723059)
S3 storage is exactly what modern databases like Snowflake work on. It’s ridiculously easy to turn compressed unstructured (say, json) data into a SQL queryable table.
Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by dcolkitt on Thursday October 08, 2020 @10:38AM (#24719449)
With the caveat that "data lake" means different things to different people, I think it's important to have a repository where the raw data exists un-manipulated in the form that it was ingested.

From the end-user standpoint that's not very useful. But that's why you have data marts that normalize the raw data into a standardized format. Ultimately though the raw data needs to remain the single source of truth. If you skip that step, and only store the post-normalized format, you're likely to run into problems down the road. This could either be because you want to change the normalization format. Or you want to use some aspect of the data that isn't captured in the normalized form. Or even you discover a pre-existing bug that affected all the previous post-ETL data.

Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by dkarl on Thursday October 08, 2020 @10:41AM (#24719480)
> Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy name.

That's kind of what I understand as well, but the data science folks pitched it in a slightly more positive way, like, "Please don't limit us just to the data you have time to nicely structure and validate. We want all of it. It doesn't matter if a column is getting truncated to three characters or columns are mislabeled or there are amounts in dollars and euros mixed together; we can still get signal out of it."

Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by freewilly1040 on Thursday October 08, 2020 @12:57PM (#24721099)
> we can still get signal out of it

Today you can, but then when the app owner drops a column (or worse, stop populating it!) in a month that signal will break, and the data lake maintainer will be in the unenviable position of navigating the completely undocumented dependency.

Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by dmpetrov on Thursday October 08, 2020 @05:50PM (#24724292)
That's might be fine. Anyway, you need to retrain your models quite often due to the concept drift. Solving data problems like this is just a part of model retraining (and concept drift).
Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by zekrioca on Thursday October 08, 2020 @10:49AM (#24719558)
More or less how a proof-of-concept for new type of wine turns out to be sold as Champagne..
Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by loopz on Thursday October 08, 2020 @11:33AM (#24720078)
As another poster said, if it's the raw data, the raw deal, it's the bets source and nice to have (but could become expensive, so better it is paid for in scale).

If it's something else, it's the antithesis of data science.

Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by detaro on Thursday October 08, 2020 @10:14AM (#24719241)
Having one central place to find and access data, even if its messy, can be better for understanding and using the data than each business unit/team/... having their own different place for (still messy) data. It can of course also be utterly pointless if done badly, not maintained, not used, ...
Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by cpard on Thursday October 08, 2020 @02:44PM (#24722327)
I think a reason that leads to the need of something like a 'data lake' or anything that looks like a messy pile of incoherent data, is the difference between how data scientists and traditionally data analysts deal with noise in the data. Most BI tasks require the data to be as clean as possible, it's important to be aware of the quality of the data before you calculate your MRR for example. On the other side, data scientists deal with noise more as a parameter of the models they are building. This difference leads to different requirements on how the pipelines should operate. In such a context, having a messy pile of data might help increase the velocity and the independence of some teams inside the company.
Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by jamesblonde on Thursday October 08, 2020 @04:20PM (#24723451)
Noise yes, null or missing values will crash your training and out-of-distribution values (which your schema will not help you with!) will be even worse, hurting performance but impossible to find.
Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by cpard on Thursday October 08, 2020 @04:52PM (#24723821)
Totally agree on that, both have still to work with data quality but in different ways and with a different mindset.
Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by sjg007 on Thursday October 08, 2020 @04:35PM (#24723646)
Data is the new oil right so you want to decouple your data consumers from the data generators. This means you dump all the data into a lake. That way a reporting team or analytics team doesn’t need to connect to your production database.
Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by xapata on Thursday October 08, 2020 @10:14AM (#24719243)
Nah, it's a messy pile of data that you want to understand rather than throw away.
Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by ipiz0618 on Thursday October 08, 2020 @11:06PM (#24726343)
In large organizations it is common to have multiple databases maintained by many different people. When a DS needs some data he/she needs to talk to all these different people, wait for them or their managers to grant access, and join the inconsistent labels. This takes a long time and some people are extremely protective of their (messy anyways) tables, requiring higher ups to straight up demand them to grant access.

Having nicely organized data is perfect, but I'd rather fetch the data myself from a pile of messy data instead of dealing with all these organizational nightmare.

Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by laichzeit0 on Thursday October 08, 2020 @12:44PM (#24720910)
What you’re referring to is sometimes called a “data swamp”.
Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by LexSiga on Thursday October 08, 2020 @10:23AM (#24719309)
You are not wrong on the buzzword, but you are not absolutely right if you suggest that it is merely that; this is a recurring question and interrogation in that specific area.

The fact that it happens to be kinda buzz worthy is a collateral aspect: everything that answers what some people wonder and that is not yet answered plainly, is.

(and I mean, the first on the front page at this very second has : "We hacked apple" in the title.)

Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by iammru on Thursday October 08, 2020 @12:31PM (#24720733)
spot on. Data lake approach is a lazy approach. Throw a pile of garbage to the storage layer and figure out how to use it later. People just kicking the can.
Re:Machine Learning Engineer Guide: Feature Store vs. Data W... by sixdimensional on Thursday October 08, 2020 @12:10PM (#24720485)
I say this as someone who built data infrastructure before and after the invention of data lakes, and I have done it every way - the old and the new.. For the record, for a lot of scenarios, the "old ways" actually do still work fine. But there are new opportunities/possibilities too.

I really understand what you are saying... I know the hype problem, I lived it. It makes me both frustrated and sad - because the hype is annoying, there is a lot of vaporware - but there is also something real that is happening too which is part of the story of the evolution of data architecture/infrastructure. My strong advice is, being open-minded is helpful - learn and take what was good/real and leave behind the stigma/hype. Something real and useful happened in terms of architecture, so take the benefits - but of course, don't compromise on delivering real, working solutions.

Regarding "data lakehouse", I struggle with the buzzword term also, but once again, I recommend looking at what is good/real and ignoring the stigma of the buzzwords. One way of looking at it is the literal translation - a data warehouse made from the components used to make data lakes. To be honest, it is a marketing term, but it is also an architectural pattern we had even before the term existed - for example - you could use a data warehouse product such as Vertica, and back it by HDFS - guess what, there's a data "lakehouse".. and most of the big database vendors can do this trick now - a full traditional data warehouse engine sitting on top of lake storage infrastructure.

There are several "real" use cases for data lakes. Precursor architectures could be seen to be "operational data stores" [1]. Data lakes are real, they are one approach to solving some problems.

These use cases could include: 1) raw, long term storage of large volumes of diversely structured data for staging/historical purposes; 2) data discovery/exploration of this data to identify patterns/models and relationships (this is both an AI and analytics use case for power users and BI/analytics/data scientists, etc.); and 3) an opportunity to change the paradigm of traditional ETL - instead of pulling from sources, one way you look at it is, allowing many diverse/distributed sources to push their data into the lake for powering analytics, exploration, AI model building, etc. It makes sense as part of lambda/kappa architecture as well - some of the "push" in can come from streaming sources as well.

Use case #1 is very much a "data infrastructure" kind of use case that we do anyway in data warehouses - especially those that do ELT (vs. ETL) - staging databases. If you want an architecture that actually helps make some sense of such a use case of data lakes more formally, one could look into Dan Lindstedt's data vault architecture [2]. While data vault modelling doesn't necessarily require "data lakes", the "raw" part of the data vault architecture use case overlaps nicely with data lakes.

[1] https://en.wikipedia.org/wiki/Operational_data_store

[2] https://en.wikipedia.org/wiki/Data_vault_modeling