News for hackers. Stuff that matters.
Re:With the rise of AI, web crawlers are suddenly controversial by 1vuio0pswjnm7 on Sunday February 18, 2024 @10:21PM (#39425950)
"Add to that the removal of public APIs and things like RSS feeds."

You mean like this public API

https://hacker-news.firebaseio.com/v0/item/39421253.json

Or this RSS feed

https://www.theverge.com/rss/index.xml

Re:With the rise of AI, web crawlers are suddenly controversial by WirelessGigabit on Monday February 19, 2024 @01:54AM (#39426866)
Reddit recently started blocking anonymous GETs to RSS feeds.

And if you use your cookies they temp-block your account after x (like x every 30 minutes) times.

And I didn't even care that there was no content on the RSS feed, I just wanted the title / notification so I don't have to use ... (I can't put anything in here as there is no alternative that allows me to track everything).

Re:With the rise of AI, web crawlers are suddenly controversial by 1vuio0pswjnm7 on Monday February 19, 2024 @03:21AM (#39427296)
Another one. This one has a rate limit.

  curl -si40A "" https://old.reddit.com/new.rss > 1.json

  sed 10q 1.json

   HTTP/1.1 200 OK
   content-length: 40582
   content-type: application/atom+xml; charset=UTF-8
   x-ua-compatible: IE=edge
   x-frame-options: SAMEORIGIN
   x-ratelimit-remaining: 93
   x-ratelimit-used: 3
   x-ratelimit-reset: 96
   x-reddit-pod-ip: 10.104.158.170:80
   x-reddit-internal-ratelimit-rls-type: ip-standard
In theory, APIs to retrieve public information suck because they are designed to be rate-limited or subject to quotas. Whereeas IME public-facing websites are much less likely to set rate limits and enforce quotas.

I prefer to use the public-facing website instead of "Web APIs" and convert HTML to SQLite which I prefer over non-LD JSON. If the public-facing website changes their HTML, I change the code in the filter. This is rare. It is very simple for me to do.