Re:With the rise of AI, web crawlers are suddenly controversial
by zarzavat on Sunday February 18, 2024 @12:52PM
(#39421253)
Add to that the removal of public APIs and things like RSS feeds. I would much rather use an API than scrape you, I will even pay a small fee, but if you don’t provide anything, then you’re getting scraped.
[ Parent
| on HN ]
Re:With the rise of AI, web crawlers are suddenly controversial
by 1vuio0pswjnm7 on Sunday February 18, 2024 @10:21PM
(#39425950)
"Add to that the removal of public APIs and things like RSS feeds."
You mean like this public API
https://hacker-news.firebaseio.com/v0/item/39421253.json
Or this RSS feed
Re:With the rise of AI, web crawlers are suddenly controversial
by WirelessGigabit on Monday February 19, 2024 @01:54AM
(#39426866)
Reddit recently started blocking anonymous GETs to RSS feeds.
And if you use your cookies they temp-block your account after x (like x every 30 minutes) times.
And I didn't even care that there was no content on the RSS feed, I just wanted the title / notification so I don't have to use ... (I can't put anything in here as there is no alternative that allows me to track everything).
Re:With the rise of AI, web crawlers are suddenly controversial
by 1vuio0pswjnm7 on Monday February 19, 2024 @03:21AM
(#39427296)
Another one. This one has a rate limit.
curl -si40A "" https://old.reddit.com/new.rss > 1.json
sed 10q 1.json
HTTP/1.1 200 OK
content-length: 40582
content-type: application/atom+xml; charset=UTF-8
x-ua-compatible: IE=edge
x-frame-options: SAMEORIGIN
x-ratelimit-remaining: 93
x-ratelimit-used: 3
x-ratelimit-reset: 96
x-reddit-pod-ip: 10.104.158.170:80
x-reddit-internal-ratelimit-rls-type: ip-standard
In theory, APIs to retrieve public information suck because they are designed to be rate-limited or subject to quotas. Whereeas IME public-facing websites are much less likely to set rate limits and enforce quotas.I prefer to use the public-facing website instead of "Web APIs" and convert HTML to SQLite which I prefer over non-LD JSON. If the public-facing website changes their HTML, I change the code in the filter. This is rare. It is very simple for me to do.