> Spend 12 months building ETLs

Okay, sounds reasonable enough for a complex enterprise.

> to feed a torrent of raw data to your lake

Well, there's the problem. Why is it taking a year to export data in its raw, natural state? The entire point of a data lake is that there is no transformation of the data. There's no need to verify the data is accurate. There's no need to make sure it's performant. It's just data exported from one system to another. If the file sizes, or record counts match, you're in good shape.

If it's taking a year to simply copy raw data from one system to another, the enterprise has deeper problems than architecture.

If you are "export[ing] data in its raw, natural state" then haven't you lost the isolation benefits of microservices? Now you have external systems dependent on your implementation details, and changing your schema will break them.

That's a problem for the future data engineers to deal with. The data lake is an insurance policy so you only need to think about these problems if you later want the data. If you already know you want to analyze the data, then a data lake is not a good choice.

Yes, it makes life harder for the data engineers in the future, but it might turn out that analysts only ever need 5% of the data in the lake, and dealing with these schema changes for 5% of the data is easier than carefully planning a public schema for 100% of it.

It can be helpful to include some small amount of metadata in the export though, with things like the source system name, date & time, # of records, and a schema version. Schema version could easily be the latest migration revision, or something like that.

The data lake is also a real, live GDPR PII time bomb if you worked out how to get the data in but not take it out

But if I haven't spent the effort to extract it, do I really own it? Let me argue that I don't have it because all my implemented queries turn up none of your data. You wouldn't tax me on gold that hasn't yet been extracted, would you? (End of joke.)

  But if I haven't spent the effort to extract it, do I really own it? 

If you collected it, you are responsible for it.

What I think you'd typically do is put different data under different keys/paths, so that red is personally identifiable data, yellow contains pointers to such data, and green is just regular data. You could have a structure like s3://my-data-lake/{red|yellow|green}/{raw|intermediate}/year={year}/month={month}/day={day}/source={system}/dataset={table}

Then you just don't keep red data for longer than 30 days.

Changing the schema of an upstream data source almost always breaks or requires updates to the downstream analytics system. It's an unavoidable problem whether its a microservice or a monolith; you just get to choose where you put the pain.

Consider:

Source Data -> Data Lake -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries

vs

Source Data -> Data Lake -> User Queries

vs

MonolithDB -> User queries

vs

MonolithDB -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries

A schema change in the source data should be easily updated in the ETL process in example 1. Most changes are minimal (adding, removing, renaming columns). And for a complete schema redesign in the source data, a new entry in the data lake should be created and the owners of the ETL process should decide if the new schema should be mangled to fit their existing reporting tables or to build new ones. Across the four models I outlined above, the first is by far the easiest to update and maintain, IMO.

If the benefit of microservices is primarily an organizational one around ownership, Conway's Law, and so on, then Example 1 still seems problematic, because it's likely a different team that has to deal with the fallout.

Another strategy is that the service has an explicit API or report specification; that way, the team that owns the services also owns the problem of continuing to support that while changing their internal implementation.

Of course, whether the benefits are worth the cost is probably organization specific, just like microservices in general.

The key insight and definition of a data lake was pulling data and storing in it's raw form form across the enterprise.

https://en.wikipedia.org/wiki/Data_lake

The reasons for it are hard to explain succinctly in HN comment but if you look up data lake there will be a lot of explanation. But it basically comes down to "is it better for the data integrators or the data consumers to massage the data?" And data lakes was the insight it's really great when the consumers decide how to massage the data.

This implies that the engineers who lobbied so hard for microservices were even all that concerned with these benefits to begin with, and took this into account when designing the architecture of the system. More often than not, in my experience, the developers involved are more concerned with code ownership than genuine architectural concerns.

Yeah I think you'd want the microservice to expose an bulk export API point, to an API specification. It might need to transform data most likely and possibly ignore some data. And then you grab data from those and piss them into the lake. The lake now conforms to your published APIs.

To me this sounds great. And honestly you should do the same thing with a monolith. Nothing worse than "oh you can't make that schema change because a customer with a read-only view will have broken reports".

You also lose benefits of microservices if they have to stop and change the data exporting system all the time too, slowing down development.

The main benefit of a dumb copy is that the production service is not impacted by reporting, only a copy is. This relates to performance (large queries) but also implementation time.

One way to avoid this is to have the microservices to publish data changes to some other (monolith) system, like an MQ system with a specified scheme for the payload.

On the other hand, the notion that “microservices == completely independent of everything else” is an unrealistic one to hold.

Where I was a bit more than a year ago they hired a real expensive consultant who did a data lake project which wasn't finished by the time my team were told to use it, so we rolled our own according to that team's instructions and best practices (best practices were a huge deal at this company, everything we did was best practices). We exported data s3 under a particular structure and I built an ETL system around that and spotify's Luigi. Noone else on my team knew what ETL was, which made me feel old. We spent two, maybe three months on this. The BI team got their data and so did the marketing automation team.

But yeah, it's funny how these projects get complicated in larger organizations. Personally I would have rolled something even simpler on gnu/posix tools and scripts, in rather less than a month.

What does Google Analytics have to do with datalakes? Are you talking about some specific scenario?

Google Analytics is associated with gleaning useful, actionable insights from your users' behavior on your web sites and in your apps, which was what the guy who sold you on the concept of a data lake was promising.

While this is absolutely true in my experience and Google Analytics has handled most of my needs in contrast with a homegrown data lake or ETL, there's always the spectre of Google pulling the rug out from under you with service shutdown or massive price increase.

Use off the shelf stuff but be prepared to have to move in a (relative) hurry.

"Google Analytics" is, as I understand the original context, shorthand for "something simple, cheap, and immediately useful." Feel free to substitute Mixpanel or some Show-HN-Google-Analytics-replacement Docker image or whatever.

Hook Microstrategy up to your lake then. If they just want inbound analytics and conversions then IT were probably recommending a data lake out of their own want to make it, rather than the actual need.

Who besides your hypothetical salesman is arguing that the raisin d’etre for data lakes is user web analytics?

In my experience people arguing for data lakes talk a lot about some unspecified future benefit. Companies typically want to learn things about their interactions with their customers that allow them to make more money, and thus GA or its equivalent represents the 80/20 — or more likely 80% of the benefit for 1% of the cost — solution.

Your statement implies a lot of assumptions about a business’s model. Our company cares about user interactions in our app, software development metrics, quality metrics, sales metrics, etc. GA is just one small piece of the puzzle.

“Data lake” may not be the right answer, but GA certainly isn’t.

I'm mostly being sarcastic but I'm partly describing an exact scenario that I'm witnessing right now. Business wants "analytics". IT starts spending a load of money. Business has no idea what it's for and buys their own thing to just track website visits which is what they wanted.