News for hackers. Stuff that matters.

A comment on Ask HN: Does Google Index “Secret” URLs Only Shared Through Gmail?.

Re:Ask HN: Does Google Index “Secret” URLs Only Shared Throu... by claudiulodro on Wednesday June 28, 2017 @12:57PM (#14655837)
You should put a robots.txt file in the S3 bucket if you don't want things in it indexed.

I (kind-of) doubt that they found the files through your e-mail, but the files in your S3 bucket are technically available for indexing since you don't have a robots.txt.

[ Parent | on HN ]
Re:Ask HN: Does Google Index “Secret” URLs Only Shared Throu... by macinjosh on Wednesday June 28, 2017 @01:03PM (#14655884)
I do have a robots.txt, but for whatever reason its not found by Google
[ Parent | on HN ]
Re:Ask HN: Does Google Index “Secret” URLs Only Shared Throu... by iurisilvio on Thursday June 29, 2017 @04:20AM (#14661612)
Are you using a custom domain? Robots.txt must be in your root folder, doesn't work with S3 urls.
[ Parent | on HN ]
Re:Ask HN: Does Google Index “Secret” URLs Only Shared Throu... by macinjosh on Thursday June 29, 2017 @12:15PM (#14664435)
yeah its the S3 URLs that are causing the problem
[ Parent | on HN ]