News for hackers. Stuff that matters.

DantesTravel

DantesTravel on Hacker News

Re:Ask HN: Have you ever used anti detect browsers for web scraping? on Friday November 18, 2022 @03:02PM (#33661220)
The browser in most case has a client for windows
Re:Ask HN: Have you ever used anti detect browsers for web scraping? on Friday November 18, 2022 @11:30AM (#33657114)
Some posts are good, more technical, others not. But in my opinion is worth reading.
Re:Ask HN: Have you ever used anti detect browsers for web scraping? on Friday November 18, 2022 @11:19AM (#33656942)
Yes, that’s what I’ve done up to now. When forced to use Playwright, I’ve noticed too that Webkit is less detected, but depends from website to website. I tried the solution described on the substack, fundamentally the gologin browser, based on chromium, opens a port on your local machine and Playwright connects to that browser, automating the crawling.
Re:Ask HN: Have you ever used anti detect browsers for web scraping? on Friday November 18, 2022 @10:44AM (#33656375)
In the substack post are mentioned several browsers, so I would say no. I didn’t even mention the Substack so no one can say I’m promoting it.
Re:Ask HN: Have you ever used anti detect browsers for web scraping? on Friday November 18, 2022 @10:42AM (#33656344)
I didn’t mention because someone would think I’m promoting it: it’s called The Web Scraping Club.
Re:Ask HN: How do I scrape my tweets without using Twitter's own tool? on Friday November 18, 2022 @08:24AM (#33654215)
I've recently discovered on a substack about web scraping the existence of anti detect browsers. Using Selenium or Playwright you can automate these browsers, each can have hundreds of isolated profiles, with different fingerprints and cookies. If you add also a proxy layer, you should be ok.

Ask HN: Have you ever used anti detect browsers for web scraping?

Posted by DantesTravel on Friday November 18, 2022 @08:21AM
from the ask-hn dept.

I'm in the web scraping industry for a while and I often spend some time creating my "swiss knife" with Playwright or Selenium in case things get tough. Thanks to a niche substack I'm following, I discovered only today the existence of anti detect browsers like GoLogin and others. From what I see, they seem a good solution for small projects, but difficult to scale in larger ones for costs of...

( Read More... | 23 comments | on HN )