Ask HN: Any advice for scraping LinkedIn at 10k profiles a day?

Posted by ptrenko on Saturday December 07, 2019 @05:19PM
from the ask-hn dept.

I'm looking to scrape about 10K profiles a day for a project I'm working on.

Don't need fake accounts or anything. Just the stuff publicly accessible when not logged in.

Does anybody have experience with this? How many proxies? Residential/Datacenter? Any tips?

( 4 comments | 2 points | on HN )

Re:Ask HN: Any advice for scraping LinkedIn at 10k profiles ... by ksputana on Sunday December 15, 2019 @10:13PM (#21800546)

so far I've only been able to do something similar with proxycrawl, any other thing I tried, even with proxies I got blocked in less than 5 minutes...

[ Parent | on HN ]

Re:Ask HN: Any advice for scraping LinkedIn at 10k profiles ... by DamonHD on Saturday December 07, 2019 @05:46PM (#21733103)

Why don't you ask LI? They may have tools.

If you don't have their permssion then you are likely running up against various 'compuster misuse' and 'privacy' laws, never mind being rude by eating their resources.

[ Parent | on HN ]

Re:Ask HN: Any advice for scraping LinkedIn at 10k profiles ... by ptrenko on Saturday December 07, 2019 @06:58PM (#21733482)

Look up HiQ vs Linkedin

[ Parent | on HN ]

Re:Ask HN: Any advice for scraping LinkedIn at 10k profiles ... by mihemihe on Saturday December 07, 2019 @06:26PM (#21733333)

You are going to be blocked if you try to do that wihout going through any officialy supported, and probably paid, API

[ Parent | on HN ]