Show HN: The Internet Portal – Warp to a Random Website, Using ICANN CZDS Data

Re:Show HN: The Internet Portal – Warp to a Random Website, ... by 1vuio0pswjnm7 on Sunday September 13, 2020 @07:24PM (#24464796)

Also, there are alternative, publicly accessible ways to get most of this public zone file data now, so I am not sure that restriction in the access agreement is anything more than an historical artifact at this point.

You could use publicly available scan data for ports 80 and 443 to pare down the list of "websites".

The goal of exposing the non-popular web is worthwhile.

[ Parent | on HN ]

Re:Show HN: The Internet Portal – Warp to a Random Website, ... by 2fast4you on Sunday September 13, 2020 @10:43PM (#24466235)

You could port scan the entire IPV4 address space(minus all reserved addresses), send a GET request to everyone that responds, filter for valid HTML. It would take no more than 5 hours on a shitty PC, a lot less if you get a small aws instance.

[ Parent | on HN ]

Re:Show HN: The Internet Portal – Warp to a Random Website, ... by luckylion on Monday September 14, 2020 @07:08AM (#24468811)

Most non-major sites are on shared hosting. Without a host name, you won't get anything useful unfortunately.

[ Parent | on HN ]

Re:Show HN: The Internet Portal – Warp to a Random Website, ... by 1vuio0pswjnm7 on Monday September 14, 2020 @04:50PM (#24474980)

Most major site are on shared hosting. (Sadly)

[ Parent | on HN ]

Re:Show HN: The Internet Portal – Warp to a Random Website, ... by sberman on Sunday September 13, 2020 @07:31PM (#24464834)

Thanks, I appreciate the feedback!

[ Parent | on HN ]