How did OpenAI crawl the web so effectively without getting blocked?

Posted by BrownSol on Friday July 28, 2023 @12:27AM
from the self-post dept.

( 4 comments | 2 points | on HN )

Re:How did OpenAI crawl the web so effectively without getti... by cratermoon on Friday July 28, 2023 @12:28AM (#36903113)

It didn't. Others did it, OpenAI used it. https://commoncrawl.org/

[ Parent | on HN ]

Re:How did OpenAI crawl the web so effectively without getti... by arthurcolle on Friday July 28, 2023 @03:51AM (#36904100)

Very carefully

[ Parent | on HN ]

Re:How did OpenAI crawl the web so effectively without getti... by throwawayadvsec on Friday July 28, 2023 @11:05AM (#36907849)

I'm pretty sure scraping is easier than building a groundbreaking LLM used by 100M people

[ Parent | on HN ]

Re:How did OpenAI crawl the web so effectively without getti... by JoeyBananas on Friday July 28, 2023 @12:05PM (#36908933)

It's not hard to scrape the internet

[ Parent | on HN ]