News for hackers. Stuff that matters.

A comment on Improving Parquet Dedupe on Hugging Face Hub. Parent comment

Re:Improving Parquet Dedupe on Hugging Face Hub by ylow on Tuesday October 08, 2024 @01:44PM (#41779873)
Great question! Rsync also uses a rolling hash/content defined chunking approach to deduplicate and reduce communication. So it will behave very similarly.
[ Parent | on HN ]
Re:Improving Parquet Dedupe on Hugging Face Hub by kwillets on Tuesday October 08, 2024 @02:01PM (#41780044)
One more: do you prefer the CDC technique over using the rowgroups as chunks (ie using knowledge of the file structure)? Is it worth it to build a parquet-specific diff?
[ Parent | on HN ]
Re:Improving Parquet Dedupe on Hugging Face Hub by ylow on Tuesday October 08, 2024 @02:35PM (#41780368)
I think both are necessary. The cdc technique is file format independent. The row group method makes Parquet robust to it.
[ Parent | on HN ]