News for hackers. Stuff that matters.
Re:Scrape like the big boys by marginalia_nu on Friday November 05, 2021 @03:55PM (#29124052)
Most pages have at least a http->https redirect, many contain a lot of old links to http content.

Usually it's error pages that really drive the large redirect chains. They often have a vibe of like some forgotten stopgap put in place to help with some migration to a version of the site that is no longer in existence.

Of course you don't know it's an error page until you reach the end of the redirect chain.

There are no comments on this comment yet.