Re:Data Deduplication at Scale
by notknifescience on Friday July 19, 2019 @04:21PM
(#20482110)
We essentially do do this--duplicates are only looked for in the (project, user, time) shard. It’s certainly more than a handful of pages, but nothing extremely cost prohibitive at all. The indexer dedupes a whole shard at a time, but only if at least one duplicate exists--in the normal case, we just do nothing. Unfortunately, we also can’t control the number of duplicates that we get (since we ingest data from our customers, we can't control how good it is), and a lot of the time, the number of duplicates are fairly frequent due to suboptimal implementations of Mixpanel, making the whole shard deduping a fair tradeoff. We definitely never touch the network :slightly_smiling_face:
If the storage engine is well-designed for the data model, a duplicate check against existing data should only touch a handful of pages in the worst case, it is an inexpensive query that rarely or never touches the network (detail depending). For ingestion of records where there is no risk of duplication, presumably the bulk of the time, this is a zero overhead model as there is no duplication state to be maintained or checked. For most scenarios that create potential duplication, this model is also quite cache friendly as a practical matter.
The pathological case for this design is when you need to check every single record for duplication (e.g. cleaning up a giant offline mess of agglomerated data that may contain an arbitrary number of duplicates), but those scenarios usually don't involve real-time stream ingestion.