Re:With the rise of AI, web crawlers are suddenly controversial
by zarzavat on Sunday February 18, 2024 @12:52PM
(#39421253)
Add to that the removal of public APIs and things like RSS feeds. I would much rather use an API than scrape you, I will even pay a small fee, but if you don’t provide anything, then you’re getting scraped.
Re:With the rise of AI, web crawlers are suddenly controversial
by 1vuio0pswjnm7 on Sunday February 18, 2024 @10:21PM
(#39425950)
"Add to that the removal of public APIs and things like RSS feeds."
You mean like this public API
https://hacker-news.firebaseio.com/v0/item/39421253.json
Or this RSS feed
Re:With the rise of AI, web crawlers are suddenly controversial
by WirelessGigabit on Monday February 19, 2024 @01:54AM
(#39426866)
Reddit recently started blocking anonymous GETs to RSS feeds.
And if you use your cookies they temp-block your account after x (like x every 30 minutes) times.
And I didn't even care that there was no content on the RSS feed, I just wanted the title / notification so I don't have to use ... (I can't put anything in here as there is no alternative that allows me to track everything).
Re:With the rise of AI, web crawlers are suddenly controversial
by 1vuio0pswjnm7 on Monday February 19, 2024 @03:21AM
(#39427296)
Another one. This one has a rate limit.
curl -si40A "" https://old.reddit.com/new.rss > 1.json
sed 10q 1.json
HTTP/1.1 200 OK
content-length: 40582
content-type: application/atom+xml; charset=UTF-8
x-ua-compatible: IE=edge
x-frame-options: SAMEORIGIN
x-ratelimit-remaining: 93
x-ratelimit-used: 3
x-ratelimit-reset: 96
x-reddit-pod-ip: 10.104.158.170:80
x-reddit-internal-ratelimit-rls-type: ip-standard
In theory, APIs to retrieve public information suck because they are designed to be rate-limited or subject to quotas. Whereeas IME public-facing websites are much less likely to set rate limits and enforce quotas.I prefer to use the public-facing website instead of "Web APIs" and convert HTML to SQLite which I prefer over non-LD JSON. If the public-facing website changes their HTML, I change the code in the filter. This is rare. It is very simple for me to do.
Re:With the rise of AI, web crawlers are suddenly controversial
by Plasmoid on Sunday February 18, 2024 @02:46PM
(#39422406)
I'm out of the loop on this. What has been happening? Do you get IP blocked? Rate limited?
Re:With the rise of AI, web crawlers are suddenly controversial
by realusername on Sunday February 18, 2024 @02:50PM
(#39422438)
You do and since the rise of Cloudflare, you need their approval to create your search engine
Re:With the rise of AI, web crawlers are suddenly controversial
by oceanplexian on Sunday February 18, 2024 @03:05PM
(#39422578)
Unfortunately for them, AI is a double edged sword. AI can trivially scrape content from a screencap. And there are thousands of residential proxies that can’t be defeated even by companies like CF.
Re:With the rise of AI, web crawlers are suddenly controversial
by spencerflem on Sunday February 18, 2024 @03:09PM
(#39422622)
If you have residential proxies, you don't need AI to scrape via video, you have the HTML already.
Re:With the rise of AI, web crawlers are suddenly controversial
by oceanplexian on Sunday February 18, 2024 @03:12PM
(#39422644)
Companies spend thousands of engineering hours building elaborate mechanisms to obfuscate JS, protect APIs, apps, trigger captchas, and hide content, all of it completely powerless against the current gen of AI tools. If a human can read it, a bot can.
Re:With the rise of AI, web crawlers are suddenly controversial
by Mr_P on Sunday February 18, 2024 @04:04PM
(#39423318)
Those same companies often invest in accessibility for vision-impaired users. I'm not sure you need a screen capture to scrape content when the site is designed to be navigable with a screen reader.
Re:With the rise of AI, web crawlers are suddenly controversial
by timthelion on Monday February 19, 2024 @02:47PM
(#39433868)
Surprisingly chatgpt is not accessible via screen reader, their captcha is purely image based.
Re:With the rise of AI, web crawlers are suddenly controversial
by CyberDildonics on Sunday February 18, 2024 @10:17PM
(#39425924)
Is OCR from 30 years ago now "AI"?
The basic social contract of the web fell apart long ago when almost everyone decided that Google was the only search engine worth serving and started aggressively blocking other crawlers.