News for hackers. Stuff that matters.
Re:Llms.txt by dns_snek on Wednesday September 04, 2024 @04:52AM (#41443379)
> a crawl requesting about 2 pages per second. It was so intense [...]

Do 2 pages per second really count as "intense" activity? Even if I was hosting a website on a $5 VPS, I don't think I'd even notice anything short of 100 requests per second, in terms of resource usage.

Re:Llms.txt by BrutalCoding on Wednesday September 04, 2024 @05:10AM (#41443492)
I assumed that he meant per client. Having a limit of 2 pages a second for a single client seems like a reasonable amount to me.
Re:Llms.txt by gtirloni on Wednesday September 04, 2024 @06:36AM (#41444021)
If ypu open DevTools and visit any website these days, you'll be surprised.
Re:Llms.txt by otherme123 on Wednesday September 04, 2024 @10:40AM (#41446366)
In my scenario you request one single page from the proxy endpoint, and all other requests go straight to the static files and have no limits. I know than no human needs to request more than 1/s from the proxy, unless you are opening tabs frantically. So far, I only get praises about how responsive and quick the sites are: being harsh with the abusers means more resources for the regulars.
Re:Llms.txt by dns_snek on Wednesday September 04, 2024 @01:01PM (#41448015)
Downvoted for asking a completely reasonable question? Where am I?