Show HN: Steel.dev – An open-source browser API for AI agents and apps

Re:Show HN: Steel.dev – An open-source browser API for AI ag... by jedberg on Tuesday November 26, 2024 @11:22AM (#42247021)

Very interesting. I worry that if I use your cloud, and a lot of other people do, all of your IP addresses will get banned by all the big players. It will definitely be a fun cat and mouse game!

Related story: Way back in the day, PayPal was just getting started, and decided eBay transactions would be their perfect customer. The only problem is that eBay didn't allow scraping. So they built an entire proxy infrastructure to go around eBay's rules and scrape them.

It worked. It worked so well, eBay bought PayPal.

The side effect of this is that I got control of the PayPal proxy infrastructure since I was on the security team for both eBay and PayPal after the acquisition.

We used that proxy farm to scrape the rest of the web looking for fake eBay sites (because they would block traffic from eBay's IPs) and we had the guys who built it help us build proxy defense for eBay and PayPal.

So this could work in your favor, if you manage to constantly scrape a large target who might want to buy you. :)

[ Parent | on HN ]

Re:Show HN: Steel.dev – An open-source browser API for AI ag... by huss97 on Tuesday November 26, 2024 @12:25PM (#42247692)

That honestly sounds so fun! Who better to build your defences than your attackers ahaha

Tying into your concern, keeping IPs fresh and high quality will definitely be a balancing act as we get bigger. It'd be one today too if we were to try to offer super granular location controls because there's only so many proxies in X state, let alone X city. Currently, we get to aggregate & QA from multiple proxy providers, so our total pool is 300M+ IPs in the US and so far we've had a 99.95% rate of getting a fresh IP address in a session. So so far so good :)

As for that last point, I guess we'll see what the future has in store for us :P

[ Parent | on HN ]

Re:Show HN: Steel.dev – An open-source browser API for AI ag... by ProofHouse on Monday December 02, 2024 @04:07AM (#42294441)

you might have a captcha issue on your discord haha or something, I cannot join

[ Parent | on HN ]

Re:Show HN: Steel.dev – An open-source browser API for AI ag... by huss97 on Monday December 02, 2024 @10:42AM (#42297241)

well that's ironic ahaha just removed that for the time being, so hopefully you're good :)

[ Parent | on HN ]

Re:Show HN: Steel.dev – An open-source browser API for AI ag... by handfuloflight on Monday December 02, 2024 @05:18AM (#42294757)

Is it possible to persist an IP address if we supply the proxy credentials?

[ Parent | on HN ]

Re:Show HN: Steel.dev – An open-source browser API for AI ag... by huss97 on Monday December 02, 2024 @10:48AM (#42297307)

yea, thats totally doable with us! The rest just depends on your proxy and whether it's rotating or persistent.

With our proxy service this is built for at least the length of a session (up to 24 hours).

[ Parent | on HN ]

Re:Show HN: Steel.dev – An open-source browser API for AI ag... by asah on Monday December 02, 2024 @07:20AM (#42295463)

+1 to residential proxy providers like joinmassive.com ! #proudinvestor

[ Parent | on HN ]

Re:Show HN: Steel.dev – An open-source browser API for AI ag... by huss97 on Monday December 02, 2024 @10:48AM (#42297312)

Looks sweet, we'll check em out!

[ Parent | on HN ]

Re:Show HN: Steel.dev – An open-source browser API for AI ag... by j45 on Monday December 02, 2024 @12:54PM (#42298552)

It appears you can set your own proxies to not use their cloud.

[ Parent | on HN ]