Re:Anyone got a contact at OpenAI. They have a spider problem
by sangnoir on Thursday April 11, 2024 @01:52PM
(#40004878)
> Content you’re allowed and capable of scraping on the Internet is such a small amount of data, not sure why people are acting otherwise
YMMV depending on the value of "you" and your budget.
If you're Google, Amazon or even lower tier companies like Comcast, Yahoo or OpenAI, you can scape a massive amount of data (ignoring the "allowed" here, because TFA is about OpenAI disregarding robots.txt)
YMMV depending on the value of "you" and your budget.
If you're Google, Amazon or even lower tier companies like Comcast, Yahoo or OpenAI, you can scape a massive amount of data (ignoring the "allowed" here, because TFA is about OpenAI disregarding robots.txt)