News for hackers. Stuff that matters.

A comment on Show HN: I made tool that let's you see everything about any website.

Re:Show HN: I made tool that let's you see everything about ... by 8organicbits on Saturday June 22, 2024 @03:43PM (#40761595)
Beautiful! Thanks for open sourcing this!

I've been working on a project [1] that probably wants to become a live crawler like this, but it's currently batch based. I'm focused on RSS feeds and microformats [2]. Can you share any details on what kind of performance / operational costs you're seeing while you're on the HN front page? The fly.toml looks like $5/month could suffice?

[1] https://alexsci.com/rss-blogroll-network/

[2] https://microformats.org/wiki/Main_Page

[ Parent | on HN ]
Re:Show HN: I made tool that let's you see everything about ... by spaceship__sun on Saturday June 22, 2024 @06:54PM (#40762978)
I'm not OP; I received ~100 thousand requests being on the front page once. It was an AI app and I quickly got rate limited to GCP Vertex AI lol.
[ Parent | on HN ]
Re:Show HN: I made tool that let's you see everything about ... by SahAssar on Sunday June 23, 2024 @08:58AM (#40767151)
> 100 thousand requests

Over how long time? Even if it's just over an hour that's just under 30 rps, over a day it's a little over 1 rps.

[ Parent | on HN ]