News for hackers. Stuff that matters.

A comment on Web Scraping in 2016. Parent comment

Re:Web Scraping in 2016 by someuser18 on Wednesday August 24, 2016 @03:53AM (#12350439)
At least Ex supports torrents and also has some custom p2p software which you can run (serves content) from which data can be siphoned off.

And what is served through their website is resized. So web-scraping is an inferior approach.

[ Parent | on HN ]
Re:Web Scraping in 2016 by fake-name on Wednesday August 24, 2016 @04:33PM (#12355023)
You seem to be assuming

1. I'm scraping the resized galleries.

2. I don't have the Hath perk that makes the galleries full sized.

3. I don't have a phash-based fuzzy image deduplication system on top of all this (see https://github.com/fake-name/IntraArchiveDeduplicator). It's main purpose is to deduplicate manga (https://github.com/fake-name/MangaCMS).

[ Parent | on HN ]
Re:Web Scraping in 2016 by ycombinatorMan on Saturday August 27, 2016 @03:33PM (#12373805)
Jesus, your projects are massive. Does your job involve working on these or are these just side things?
[ Parent | on HN ]
Re:Web Scraping in 2016 by fake-name on Tuesday August 30, 2016 @04:26PM (#12393568)
It's all entirely hobby things.
[ Parent | on HN ]