News for hackers. Stuff that matters.

A comment on Show HN: PaperDelivery – fetch articles from your Twitter feed.

Re:Show HN: PaperDelivery – fetch articles from your Twitter... by harish5p on Saturday July 31, 2021 @12:05PM (#28019319)
News websites are asking if I'm a robot instead of showing the headlines.Is it common? https://postimg.cc/47xNhT1g
[ Parent | on HN ]
Re:Show HN: PaperDelivery – fetch articles from your Twitter... by lonesword on Saturday July 31, 2021 @03:14PM (#28020834)
Unfortunately yes. They see too many requests from paperdelivery.co and block requests originating from that domain - they only want the google bot to crawl their site :) I think I have two options:

1. Respect their decision to not allow PaperDelivery to fetch that page.

2. Use a set of proxies and headless browsers or outright impersonate the Google bot to trick the news websites into allowing PaperDelivery to fetch that page.

[ Parent | on HN ]