News for hackers. Stuff that matters.
Re:Emerging Architectures for Modern Data Infrastructure by ethanwillis on Saturday October 17, 2020 @10:12PM (#24815037)
While this is an article about data infrastructure I feel like we're missing the forest for the trees.

What is most important here in my opinion is that the underlying data is useful. If your underlying data wasn't collected, collected properly, or even worse the wrong data was collected.. then setting up data infrastructure will be a boondoggle that will cause your organization to be data hostile.

Just as much, if not more effort, needs to go into collecting the right data in the right way to fill your data infrastructure with. Most of the projects I've seen or heard of are just people taking the same old data that Ted in accounting, Jill in BI, etc. are already pretty proficient at using. So the gains you get by moving that into a modern infrastructure are marginal. How many more questions can you really ask of the same data that people have decades of experience with and an intuitive sense for?

Re:Emerging Architectures for Modern Data Infrastructure by teej on Saturday October 17, 2020 @11:28PM (#24815319)
The biggest shift has been towards data lake (store everything) away from data cubes (store aggregates). This makes it orders of magnitude easier to diagnose, debug, and assert the correctness of data.

So these trends aren’t in a vacuum, they directly support the issues you discuss.

> Most of the projects I've seen or heard of are just people taking the same old data ...

I don’t disagree with you here. But in my experience it’s about getting Frank in marketing to use the same numbers as everyone else.

When you have 5 different ads platforms that all take revenue credit for a single conversion and have conflicting attribution models, and none of them add up to what accounting says is in the bank account. That’s a hairy problem.

There are different flavors of that class of problem at lots of companies.

Re:Emerging Architectures for Modern Data Infrastructure by altdatathrow on Sunday October 18, 2020 @12:25AM (#24815542)
> The biggest shift has been towards data lake (store everything) away from data cubes (store aggregates).

I don't think this is any shift. The "store everything" has always existed in my experience, that's how the aggregates were built in the first place. The aggregates were for speed and convenience, and you drill-down as necessary, including to the individual record level.

Maybe the shift is people thinking that it's cheaper to just analyze the entire corpus on-demand because we can throw a spark cluster at it?

Re:Emerging Architectures for Modern Data Infrastructure by hestefisk on Sunday October 18, 2020 @01:43AM (#24815757)
I agree, data warehouse was what the data lake is today. Data cube is the aggregation of data in the warehouse, and then you can drill down and roll up. Difference between warehouse and lake is the emphasis on correctness (one canonical data model) and deduplication of data (when warehouses were invented, storage was expensive so one tried to normalise it into a star schema with as little duplication as possible — when emergence of cheap storage, this is less important and we can spend less time developing fancy ETL processes to make everything fit into one, conformant data model).
Re:Emerging Architectures for Modern Data Infrastructure by altdatathrow on Sunday October 18, 2020 @01:57AM (#24815805)
> this is less important and we can spend less time developing fancy ETL processes to make everything fit into one, conformant data model

And that's precisely why modern data processes are inferior to 20 years ago. People reinvent the wheel over and over and spend massive budgets on unnecessary tech stacks that would be alleviated if the time was simply taken to model the data.

A clean data model is about a whole lot more than simply storage space.

Re:Emerging Architectures for Modern Data Infrastructure by hestefisk on Sunday October 18, 2020 @09:06AM (#24817597)
Agree if people would model everything to a “Grand Unified Data Model” of everything, it would be a lot more efficient... unfortunately that is very hard and puts a massive bureaucracy around data governance and management. It slows things down. I guess a more modern approach is to relax those constraints a bit and realise that some data can be expressed in different ways, and that duplication isn’t too much of a concern because storage is cheap. That said, it’s not an excuse for reinventing the stack over an over. I think the thirst to reinvent has largely been driven by the shift from expensive proprietary solutions like Teradata and Oracle to open source ones. That’s a positive shift.
Re:Emerging Architectures for Modern Data Infrastructure by haddr on Wednesday October 21, 2020 @10:26AM (#24848178)
And this is why big enterprises are moving out of data warehouse model for processing all the data and prefer a data lake concept. It is better to centralize some aspects of the data, but definitely not all (like common relational model for all sources, as it is in data warehouse). The data warehouse model quickly becomes super expensive to maintain and evolve.
Re:Emerging Architectures for Modern Data Infrastructure by dima_vm on Sunday October 18, 2020 @04:51AM (#24816411)
You miss the critical difference -- nowadays people don't store aggregates, they just scan sharded data very fast. That simplifies a lot of things, because you don't need to keep two databases in sync (raw and aggregated).
Re:Emerging Architectures for Modern Data Infrastructure by gfodor on Saturday October 17, 2020 @11:44PM (#24815383)
I was working in data analytics + data science a decade ago and we stored everything, not aggregates, and pushed them through hadoop. I have been "out of the game" since then. What has changed that is making people saying "store everything" is a new phenomenon? (genuine question bc I am clearly missing something.)
Re:Emerging Architectures for Modern Data Infrastructure by teej on Saturday October 17, 2020 @11:58PM (#24815435)
It’s not a new phenomenon so much as it has emerged as an important shift from the status quo 20 years ago.

What’s changed in the last 10 years are the access patterns. There’s increased demand to have arbitrary query access over the raw data. The most impactful technology changes have been about pushing the access layer (queries, stream & batch processing, dashboards, BI tools, etc) down as close to the raw data as possible and making that performant. What’s fallen out of that are better MPP OLAP databases (snowflake), new columnar formats (parquet), SQL as the transform layer (dbt).

Re:Emerging Architectures for Modern Data Infrastructure by gfodor on Sunday October 18, 2020 @01:17AM (#24815688)
Ah that makes sense. Thanks.
Re:Emerging Architectures for Modern Data Infrastructure by fscherer on Sunday October 18, 2020 @07:41AM (#24817127)
why is it actually that SQL "re-emerged" as the transformation layer? I thought that it first shifted from SQL to Query Builders inside talend, matillion etc. Why now SQL again?
Re:Emerging Architectures for Modern Data Infrastructure by scrollbar on Sunday October 18, 2020 @09:18AM (#24817674)
Probably just the emergence of dbt? I’ve only been doing ETL for a couple years personally but couldn’t imagine using so much SQL in our pipelines without a framework like dbt
Re:Emerging Architectures for Modern Data Infrastructure by goBackwards00 on Sunday October 18, 2020 @12:31AM (#24815562)
The problem of data confusion you describe is resolved by replacing management. That’s not an engineering issue that requires new technology (consider the source of social power for author; selling technology).

That’s an engineering issue that needs new engineering management who don’t enable wasting company resources making incompatible APIs in the first place.

We already did the monolithic DB design, I used to name those hosts “ocean”. And we already know the math. “Data lake” is just more jargon by a salesman to obfuscate peddling the same old abstraction, and wow fresh grads with new words for hyping the same old habits.

While not the author of this piece, Bezos is quoted as pointing out how circular social behavior is.

What do you think the odds of this author being on a similar page?

Have humans evolved much in 100 years? Or does the con simply get rewritten for the next generation to hide a simple truth?

What’s keeping people going in this circle isn’t logistical necessity. It’s us.

Re:Emerging Architectures for Modern Data Infrastructure by huy on Saturday October 17, 2020 @11:36PM (#24815354)
I think you have a point, but there are more nuiances than that.

There are typically 2 types of data to collect: Transactional data and behavioural data.

Most transactional data, due to their important nature, are already generated and captured by the production applications. Since the logic is coded by application engineer, it's usually hard to get this data wrong. These data are then ETL-ed (or EL-ed) over to a DW, as described by the article.

For behavioural data, this is where your statement will most apply to. This is where tools like Snowplow, Posthog, Segment, etc come in to set up the proper event data collection engine. This is also where it's important to "collect data properly", as these kinds of event data changes structure fast, and hard to keep track over time. I'd admit this space (data collection management) is still nascent, with only tools like iterative.ly on the market.

Re:Emerging Architectures for Modern Data Infrastructure by NightMKoder on Sunday October 18, 2020 @01:32AM (#24815726)
I completely agree - there's only so many ways to slice the data. The caveat is - the type of data matters quite a bit for the data architecture. There's another thread that mentions sensor data as a source of complexity since the data has a theoretical delay between events (i.e. period) of 0 - something few systems are built to handle, even if you sample approximations at some fixed frequency. Algorithmic trading is a similar domain that still has a huge bar for entry - a sign that _this isn't easy_.

The fidelity of the data is of course important, but I would claim it's not a blocker. Yes, you need to trust the data you collect. That's table stakes - if you can't collect data correctly at all, even without worrying about the past, you're in for a world of hurt. It's P0. That said, a lot of people assume you also need to do this historically - and that's not the case - at least for ML.

Reinforcement learning has been making great strides in recent years. If you're in this situation - you have a flow where you want to use a model without having any past data to train with - use something like VW's contextual bandits [1]. You don't need historical data to build your model, just real-time decision point & reward signals. Once deployed, the model converges over time to the optimal model using real-time feedback.

All that said - baby steps are important. If you're in this situation, start by getting fidelity and then expand scope slowly without sacrifice to fidelity. It's a lot easier to backfill than to "fix" data - get that right and it get's easier from there. You'll need fixups regardless - mistakes happen and requirements change - but you have to start with something you trust, at least in the moment it's deployed.

[1] https://vowpalwabbit.org/tutorials/contextual_bandits.html

Re:Emerging Architectures for Modern Data Infrastructure by an_opabinia on Saturday October 17, 2020 @11:02PM (#24815212)
Is there any evidence that the vast amounts of clicks and user interactions companies have been collecting are worth anything at all?

Let’s say I deleted every time series whose Y axis isn’t measuring US dollars in every tech company’s database everywhere. Maybe for all those time series you just store the most recent value. Describe to me what would be lost.

You’re onto something but you’re not going far enough! Most, if not all, historic metadata, analytics and behavioral data collection - when it is not measuring literal dollar amounts - is completely worthless.

Re:Emerging Architectures for Modern Data Infrastructure by mgraczyk on Saturday October 17, 2020 @11:11PM (#24815247)
This is completely wrong, and nobody who works with data at any scale could possibly believe anything like this.

We literally run long term A/B tests with thousands of variations of what you're describing. The purpose of these tests is to measure the effect of losing some data. The tests show (to nobody's surprise) that each piece of data is useful. These tests tell us exactly how useful each piece of data is.

Honestly when I read comments like this I have to wonder, do you really believe that thousands of companies spend trillions of dollars a year for something that doesn't work? Maybe talk to somebody who works on this stuff a bit?

Re:Emerging Architectures for Modern Data Infrastructure by abernard1 on Saturday October 17, 2020 @11:20PM (#24815289)
> Honestly when I read comments like this I have to wonder, do you really believe that thousands of companies spend trillions of dollars a year for something that doesn't work?

Joking/not-joking. Have you ever been to the Bay Area?

Yes. Emphatically yes it is the case companies spend trillions of dollars unnecessarily.

We've seen this with people who didn't know how to build microservices and farcical "LMNOP" [1] type services that might as well be a joke. We've seen it with gigantically-valued unicorns that over-engineered tons of crap and hired too many people and still can't make a profit. We've seen it with CMOs and massively overpriced marketing technology because budgets and statuses are related. We'll see it with tons more iterations of this exact same affluenza.

The history of our industry is that the margins on software are so good that people can afford to do crazy nonsense.

[1] https://www.youtube.com/watch?reload=9&v=y8OnoxKotPQ

Re:Emerging Architectures for Modern Data Infrastructure by mgraczyk on Saturday October 17, 2020 @11:55PM (#24815427)
Yeah I mean this misses the main argument I'm making.

I have vast amounts of firsthand evidence from randomized controlled trials that non-financial data can be used to create value. This is enough evidence for anyone in the industry.

Presumably the commenter doesn't have access to this evidence. Instead he has to rely on other heuristics, like the weaker argument that companies spend trillions on data and analytics.

Companies sometimes waste money, and maybe microservices are an example of this. But companies collectively spend 3-4 orders of magnitude more money on "all data that does not have USD units" than on microservices, so the commenter should take that as strong evidence that data can be used to create value.

Re:Emerging Architectures for Modern Data Infrastructure by abernard1 on Sunday October 18, 2020 @12:11AM (#24815486)
His quote was:

> Is there any evidence that the vast amounts of clicks and user interactions companies have been collecting are worth anything at all?

And for the huge majority of companies, it is not. Even many 1B+ dollar companies. Most value produced by businesses still could exist with boring ETL or BI concepts that have been around forever, because the hardware powering it is so fast. Many if not most of those businesses probably would be better off.

So yes, companies blow a lot of money on stuff with questionable ROI. I don't discount your experiences and that there are cases where the complexity might have a payoff. But honestly, we've seen periods of excess complexity and waste in software before: it's the norm.

Re:Emerging Architectures for Modern Data Infrastructure by dcgudeman on Saturday October 17, 2020 @11:46PM (#24815390)
Just so I don't misunderstand your point: "Yes, companies without coordinating are collectively spending vast sums of precious resources on a product/service, BUT they are all misguided and wasting their money because look at these unrelated events where individuals misplaced their efforts/ money". Is that correct?
Re:Emerging Architectures for Modern Data Infrastructure by abernard1 on Saturday October 17, 2020 @11:58PM (#24815436)
> wasting their money because look at these unrelated events where individuals misplaced their efforts/ money

They're not unrelated. They're all related to fat 60-80% profit margins on SaaS. And no coordination is necessary to spend money on silly, make-work activities if you have margins like that.

Software has extremely low, borderline zero, variable costs. A lot of these companies that spend money on examples like I've given probably could hire nobody at all and still have crazy growth because of the unit economics. (Not coincidentally, the companies most in want of "Big Data" solutions tend to be past this point.)

I can get 1M QPS on a silly Aurora setup with replicas. Best tool for all jobs: no. But don't tell me that dollar-for-dollar a data architecture with like 25 different components is dramatically superior to an OLTP db, OLAP offline store + batch jobs, and a streaming system.

Re:Emerging Architectures for Modern Data Infrastructure by Sevii on Sunday October 18, 2020 @02:07AM (#24815831)
That video is great, I've actually worked on a project where we tried to federate customer databases across 3+ mergers. Started before I joined the company and after 3-4 iterations they shipped something (that worked) around 3 years later right before I left.
Re:Emerging Architectures for Modern Data Infrastructure by an_opabinia on Sunday October 18, 2020 @12:00AM (#24815442)
I’m not trying to get into a flame war with you, there’s no reason to be hostile. It sounds like you “work on this stuff a bit,” you’re welcome to share concrete examples, it would be really interesting!

I think it’s an intriguing thought exercise. For example, does one need the entire history of interactions with e.g., an Instagram post, or just aggregated measurements? I’m not like, against measuring. Just against warehousing of non financial timeseries.

Re:Emerging Architectures for Modern Data Infrastructure by mgraczyk on Sunday October 18, 2020 @02:47AM (#24815980)
Yeah don't mean to argue, here are some examples. It's difficult for any company to be competitive at scale without sufficient logging to support all of these things.

* Timestamps of events related to content loading and rendering. This is crucial for debugging and improving load times.

* Backfilling aggregated data so that ML models can be trained without waiting weeks for new streaming aggregation.

* Answering product questions of almost any kind that weren't asked when logging was built.

Concrete example from my recent experience, you may want to know how often people like a post then later look at comments, vs look at comments then later like a post. This gives you information about cause and effect.

Re:Emerging Architectures for Modern Data Infrastructure by abernard1 on Sunday October 18, 2020 @03:58AM (#24816232)
The first one doesn't even need much historical data. Unless you have some very unoptimized periodic jobs, the last few days or something is plenty.

The second can be done simply on something like Dynamo, CosmosDB, or your cloud-hosted NoSQL of choice. Heck, it can even be done on Aurora or vanilla Postgres + partitioning if it's <64TB.

The third can be done with any off the shelf cloud data warehouse software, at many petabyte scale. And even then, I'm sorry, but I just don't believe you that the product clicks over some large timeframe are historically relevant if your software and UI changes often.

All of these things mentioned have had extremely simple, boring solutions at petabyte scale for >10 years, and in some cases more than that. If you add a batch workflow manager and a streaming solution like Spark, that's like 3-4 technologies total to cover all these cases (and many more!)

Re:Emerging Architectures for Modern Data Infrastructure by gfodor on Saturday October 17, 2020 @11:47PM (#24815394)
yes consider RPC:

CORBA

EJBs

SOAP

Microservices <--- You are here

Re:Emerging Architectures for Modern Data Infrastructure by Supermancho on Sunday October 18, 2020 @12:32AM (#24815564)
> Is there any evidence that the vast amounts of clicks and user interactions companies have been collecting are worth anything at all?

Yes. Every advertising platform ever uses this information. In Europe, you have to have regulation that makes account costing (what the US might call forensic accounting) possible. The presentations on A/B tests by FANG companies might also interest you. They are on Youtube.