News for hackers. Stuff that matters.
Re:Ask HN: Identifying duplicate data from a large dataset? by verdverm on Monday March 18, 2019 @04:02PM (#19424640)
This is easier than deduplicating the many different URLs which have the same content. A harder problem awaits you!

ML & basic stats

Re:Ask HN: Identifying duplicate data from a large dataset? by gerenuk on Monday March 18, 2019 @04:29PM (#19424917)
What approach would you be going for initially for deduplicating same urls?
Re:Ask HN: Identifying duplicate data from a large dataset? by verdverm on Tuesday March 19, 2019 @04:15AM (#19428967)
You might look at a real data processing system, like something from the Apache projects