Re:Search all Craigslist cities for a remote job
by gerpsh on Monday April 08, 2019 @06:39PM
(#19609802)
Dude, you're making ScraperAPI requests (with your API key in plain text as a parameter) directly from the client app...this thing is a nice idea but it'd probably be a good idea to clean it up and repost.
Re:Search all Craigslist cities for a remote job
by smt88 on Monday April 08, 2019 @07:03PM
(#19609988)
Craigslist is famously aggressive about scraping, too. While helping with some university research, I found that their anti-scraping defenses were really difficult to get around.
They've also been litigious about this in the past, so this project would undoubtedly get a cease-and-desist if it ever became a blip on their radar.
Re:Search all Craigslist cities for a remote job
by CalRobert on Tuesday April 09, 2019 @04:09AM
(#19612688)
Which was a shame, because padmapper stuck it on a map and linked to Craigslist, instead of just taking the content. If Google Maps did it nobody would think twice.
Re:Search all Craigslist cities for a remote job
by stevenwoo on Tuesday April 09, 2019 @04:23AM
(#19612757)
I was a contractor at a company where one of the divisions did Craiglist scraping as part of the business model and they had a closet full of laptops doing the scraping part of the job - thirdhand what I heard was separate PC's were necessary for the anti-scraping workaround they were using.
Re:Search all Craigslist cities for a remote job
by everdev on Monday April 08, 2019 @09:22PM
(#19610808)
Proof of Concept is a vague term and doesn't give you any sort of protection or defense. If it's against Craigslist's terms you're taking on some risk by making and public and promoting it.
Re:Search all Craigslist cities for a remote job
by ravenstine on Monday April 08, 2019 @08:54PM
(#19610640)
That's basically what happened to Padmapper, sadly.
With the amount of scams causing a decline of their platform, I'd think that Craigslist would want other sources of traffic. I know that most sites have a policy against scraping/indexing, but Craigslist has their head up their own ass.
Re:Search all Craigslist cities for a remote job
by biztos on Tuesday April 09, 2019 @05:38AM
(#19613047)
Maybe timeliness is the important thing to Craigslist.
If you scrape all the jobs data as soon as it shows up, and repost it to your site, then maybe employers think you're the one they should be paying, based on prospects coming in your site and not CL.
I would assume that matters more than if you scraped older data or got older data by scraping it more slowly.
Re:Search all Craigslist cities for a remote job
by tomca32 on Monday April 08, 2019 @07:03PM
(#19609985)
Cool idea, but the implementation is kinda crazy. Good enough to just test the idea at home on your own machine, but pretty risky to expose it to the world and let a ton of other people go nuts making bazzilions of requests.
Re:Search all Craigslist cities for a remote job
by rkho on Tuesday April 09, 2019 @01:27AM
(#19612036)
You should really consider caching results and only fetching them again after a certain amount of time has passed since the last time you scraped them. This way you're not blowing through your ScraperAPI limit.
Re:Search all Craigslist cities for a remote job
by timsayshey on Tuesday April 09, 2019 @11:12AM
(#19615428)
This is an amazing project! Probably one of the most useful ways to actually find quality job listings. If only Craigslist would offer site-wide search natively.
Btw, you should open-source this on GitHub. You should also include a crypto/paypal donation option.
Re:Search all Craigslist cities for a remote job
by mothsonasloth on Tuesday April 09, 2019 @07:34AM
(#19613557)
Its cool but the main problem I am facing with searching for remote jobs. Is that most remote US jobs are not technically remote. They are for US citizens only, so I can't work on it from Europe.
Re:Search all Craigslist cities for a remote job
by anonytrary on Monday April 08, 2019 @10:05PM
(#19611073)
I, too, entered a search query, pressed go, and witnessed absolutely nothing happen. Turns out, you just have to wait a bit. It scrapes Craigslist in real time from the client, so you have to wait a bit for results to pop up. It'd be nice if they had a loading/progress indicator.
Re:Search all Craigslist cities for a remote job
by sithlord on Tuesday April 09, 2019 @08:07AM
(#19613754)
would this not fall into a similar category of happened to linkedin? The data is publicly facing, and not behind a paywall or a password, therefore, basically fair-use of internet?
ianal, nor do I know much about the subject, but seems like if its not at a minimum password protected information, it is pretty fair-use.
They've also been litigious about this in the past, so this project would undoubtedly get a cease-and-desist if it ever became a blip on their radar.
Not too hard to see where the fulcrum of this "settlement" is: https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-se...
What sort of defenses are they using?
Then it sent 503 XHR requests:
https://i.imgur.com/R9D23iU.png
With the amount of scams causing a decline of their platform, I'd think that Craigslist would want other sources of traffic. I know that most sites have a policy against scraping/indexing, but Craigslist has their head up their own ass.
If you scrape all the jobs data as soon as it shows up, and repost it to your site, then maybe employers think you're the one they should be paying, based on prospects coming in your site and not CL.
I would assume that matters more than if you scraped older data or got older data by scraping it more slowly.
Also, exposed API key in the request.
console.log(items);
console.debug(items);
Btw, you should open-source this on GitHub. You should also include a crypto/paypal donation option.
"You're sending requests a bit too fast! Please slow down your requests, or contact [email protected] with any questions."
ianal, nor do I know much about the subject, but seems like if its not at a minimum password protected information, it is pretty fair-use.