News for hackers. Stuff that matters.

A comment on Yahoo is making a return to search. Parent comment

Re:Yahoo is making a return to search by staplung on Monday January 30, 2023 @10:45PM (#34590667)
If they can't hit `/*/tree` is there a way to know the URLs of the files?
[ Parent | on HN ]
Re:Yahoo is making a return to search by pancrufty on Monday January 30, 2023 @10:57PM (#34590792)
Direct links from crawlable pages
[ Parent | on HN ]
Re:Yahoo is making a return to search by kadoban on Monday January 30, 2023 @10:54PM (#34590744)
Sure, clone the git repo.
[ Parent | on HN ]
Re:Yahoo is making a return to search by utopcell on Tuesday January 31, 2023 @05:42AM (#34593576)
GitHub would not be happy with Google cloning all repos, and many of them at a high frequency, in order to circumvent a robots.txt restriction.
[ Parent | on HN ]
Re:Yahoo is making a return to search by kadoban on Tuesday January 31, 2023 @07:54AM (#34594466)
They're clever people, they could just do partial updates (pull instead of clone). I doubt it'd be that much of a strain.
[ Parent | on HN ]