Re:Twitter now requires an account to view tweets
by qingcharles on Saturday July 01, 2023 @12:00PM
(#36551500)
I don't expect HN give a fuck about the scraping. It's pure HTML, no images, probably cached all to hell for users who aren't logged in anyway.
The one thing I see as a future issue is that people are starting to post comments that clearly look like they were manufactured by ChatGPT and friends. Or that could just be the way some people talk and I've spent too long with ChatGPT now and start to smell it everywhere.
Re:Twitter now requires an account to view tweets
by dredmorbius on Saturday July 01, 2023 @01:48PM
(#36552823)
HN does have performace / capacity issues, and you'll find that if you're crawling the site rapidly, you'll quickly find your IP banned.
I've had that happen even under manual browsing (when logged out). My front-page analytics project hit that limit quickly (within about 30 request, probably less). Adding in a reasonable delay got around that.
Keep in mind that a lot of Web infrastructure tends over time to operate just at the edge of stability, as capacity costs money.
If the world around HN (including its community) changes stasis can damage or kill it as well.
Specifically regarding the issue of the original posting:
- HN is already an important data source for large language model training.
- To the best of my knowledge there is no freely downloadable and current data dump of HN.
- The HN-API does not offer all the data that scraping can get. For example if a post had ever hit the front page or the highest front page position reached is an interesting data point that is missing.
- The Algolia-HN-API has the same limitations.
In my opinion this will lead to increased usage of the API and increased scraping which all costs money. HN might be forced to find a solution for this.