The Cost of Being Crawled: LLM Bots and Vercel Image API Pricing

Re:The Cost of Being Crawled: LLM Bots and Vercel Image API ... by cachedthing0 on Monday April 14, 2025 @08:40PM (#43687852)

"Together they sent 66.5k requests to our site within a single day."

Only scriptkiddies are getting into problems by such low numbers. Im sure security is your next 'misconfiguration'. Better search an offline job in the entertainment industries.

[ Parent | on HN ]

Re:The Cost of Being Crawled: LLM Bots and Vercel Image API ... by aledalgrande on Monday April 14, 2025 @09:01PM (#43687985)

I know the language earned you the downvotes (please be kind), but the author of the article is ex Google and ex AWS, I too would expect some better infra in place (caching?) and certainly not Vercel.

[ Parent | on HN ]

Re:The Cost of Being Crawled: LLM Bots and Vercel Image API ... by ilyabez on Tuesday April 15, 2025 @03:56AM (#43690130)

Hi, I'm the author of the blog (though I didn't post it on HN).

Some context here.

I was actually a PM at Google & AWS, not an engineer. Even though I have a CS degree, I had not been professionally coding for almost 20 years. This is my comeback to software development and I've got a lot to catch up on. Hope this sets the stage appropriately.

I mentioned in an earlier comment that we didn't actually build the site and it was on a back-burner until we added episode pages and got hit by costs. It's a lesson learned indeed and we're now treating the website as a first-class citizen in our stack.

[ Parent | on HN ]

Re:The Cost of Being Crawled: LLM Bots and Vercel Image API ... by aledalgrande on Tuesday April 15, 2025 @09:49PM (#43700549)

Good for you Ilya, best of luck with your company!

[ Parent | on HN ]