News for hackers. Stuff that matters.

A comment on More than 9M broken links on Wikipedia are now rescued. Parent comment

Re:More than 9M broken links on Wikipedia are now rescued by Nition on Monday October 01, 2018 @11:50PM (#18118677)
Would be nice if you could store a diff, so a changing counter would only have to store the changed counter after the initial save.
[ Parent | on HN ]
Re:More than 9M broken links on Wikipedia are now rescued by zaarn on Tuesday October 02, 2018 @03:01AM (#18119366)
You can do this if you group all site visits into one common WARC and compress it (or dedup it otherwise).

WARC itself does have a method of dedup if the response is the same (or mostly the same) but terrible if content changes.

[ Parent | on HN ]
Re:More than 9M broken links on Wikipedia are now rescued by murukesh_s on Tuesday October 02, 2018 @12:21AM (#18118769)
Wondering the same. How if used git protocol itself? Not sure how efficient is git, but if it is then it's a relatively easier change
[ Parent | on HN ]
Re:More than 9M broken links on Wikipedia are now rescued by eythian on Tuesday October 02, 2018 @08:27AM (#18120867)
Git doesn't store diffs, it stores whole files.
[ Parent | on HN ]
Re:More than 9M broken links on Wikipedia are now rescued by rakoo on Tuesday October 02, 2018 @05:30PM (#18125879)
No, git packs do store base blobs and diffs, which are not visible to end users; the git plumbing hides this and only presents blobs to the user.
[ Parent | on HN ]