News for hackers. Stuff that matters.

A comment on Ask HN: Does Google Index “Secret” URLs Only Shared Through Gmail?. Parent comment

Re:Ask HN: Does Google Index “Secret” URLs Only Shared Throu... by macinjosh on Wednesday June 28, 2017 @01:03PM (#14655884)
I do have a robots.txt, but for whatever reason its not found by Google
[ Parent | on HN ]
Re:Ask HN: Does Google Index “Secret” URLs Only Shared Throu... by iurisilvio on Thursday June 29, 2017 @04:20AM (#14661612)
Are you using a custom domain? Robots.txt must be in your root folder, doesn't work with S3 urls.
[ Parent | on HN ]
Re:Ask HN: Does Google Index “Secret” URLs Only Shared Throu... by macinjosh on Thursday June 29, 2017 @12:15PM (#14664435)
yeah its the S3 URLs that are causing the problem
[ Parent | on HN ]