Ask HN: Why aren’t the Wayback Machine archived pages indexed by search engines?

Re:Ask HN: Why aren’t the Wayback Machine archived pages ind... by cyberlab on Friday March 05, 2021 @01:59PM (#26360582)

I used an extension once in Firefox that allows you to view the archived version of any URL (providing the site allowed the WM crawler in their robots.txt). It worked for both working URLs and URLs that 404'd or didn't exist anymore / bitrotted pages.

[ Parent | on HN ]

Re:Ask HN: Why aren’t the Wayback Machine archived pages ind... by hosa on Friday March 05, 2021 @02:25PM (#26360945)

Yes I know that extension.. But my purpose is different, for researching and education , there are pages that only exist in the internet archive, and so I do not want to search the same thing two times in two different places..

[ Parent | on HN ]

Re:Ask HN: Why aren’t the Wayback Machine archived pages ind... by cyberlab on Friday March 05, 2021 @03:25PM (#26361592)

Yeah but if the site issues a 404, you simply right click and the extension will show you the archived copy. Sadly many sites fail to provide a decent 404 page if a resource doesn't exist, they just do redirect spamming or in the worst case: 503 error[0]

(But I understand your need for text search on these services)

[0] https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/503

[ Parent | on HN ]