News for hackers. Stuff that matters.
Re:Search all Craigslist cities for a remote job by gerpsh on Monday April 08, 2019 @06:39PM (#19609802)
Dude, you're making ScraperAPI requests (with your API key in plain text as a parameter) directly from the client app...this thing is a nice idea but it'd probably be a good idea to clean it up and repost.
Re:Search all Craigslist cities for a remote job by criddell on Monday April 08, 2019 @07:01PM (#19609970)
I thought these multi-region searches were against Craigslist's terms of service.
Re:Search all Craigslist cities for a remote job by smt88 on Monday April 08, 2019 @07:03PM (#19609988)
Craigslist is famously aggressive about scraping, too. While helping with some university research, I found that their anti-scraping defenses were really difficult to get around.

They've also been litigious about this in the past, so this project would undoubtedly get a cease-and-desist if it ever became a blip on their radar.

Re:Search all Craigslist cities for a remote job by MegaButts on Monday April 08, 2019 @07:29PM (#19610182)
I'm guessing that's because of AirBnB and how they got their start by essentially stealing posts from Craigslist.
Re:Search all Craigslist cities for a remote job by massivecali on Monday April 08, 2019 @07:41PM (#19610258)
I thought Padmapper got their start that way, by essentially providing a better apartment hunt interface than CL had at the time.
Re:Search all Craigslist cities for a remote job by CalRobert on Tuesday April 09, 2019 @04:09AM (#19612688)
Which was a shame, because padmapper stuck it on a map and linked to Craigslist, instead of just taking the content. If Google Maps did it nobody would think twice.
Re:Search all Craigslist cities for a remote job by giancarlostoro on Tuesday April 09, 2019 @07:36AM (#19613573)
CraigsList might make them think twice. I don't think they want to overly exhaust their own resources, what they have works for them.
Re:Search all Craigslist cities for a remote job by rhizome on Monday April 08, 2019 @10:57PM (#19611359)
And they got the banhammer, too.

Not too hard to see where the fulcrum of this "settlement" is: https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-se...

Re:Search all Craigslist cities for a remote job by Reedx on Monday April 08, 2019 @09:44PM (#19610954)
> I found that their anti-scraping defenses were really difficult to get around

What sort of defenses are they using?

Re:Search all Craigslist cities for a remote job by smt88 on Tuesday April 09, 2019 @11:46AM (#19615830)
They had fantastic detection of headless Chrome and curl requests, a thorough IP blacklist, aggressive rate limiting, and possibly some JS stuff.
Re:Search all Craigslist cities for a remote job by stevenwoo on Tuesday April 09, 2019 @04:23AM (#19612757)
I was a contractor at a company where one of the divisions did Craiglist scraping as part of the business model and they had a closet full of laptops doing the scraping part of the job - thirdhand what I heard was separate PC's were necessary for the anti-scraping workaround they were using.
Re:Search all Craigslist cities for a remote job by rhizome on Monday April 08, 2019 @10:56PM (#19611351)
All scrapey automated stuff is. I doubt this will be alive next week, or tomorrow.
Re:Search all Craigslist cities for a remote job by throwaway190102 on Monday April 08, 2019 @11:01PM (#19611380)
already appears broken.
Re:Search all Craigslist cities for a remote job by eagsalazar2 on Monday April 08, 2019 @11:37PM (#19611558)
What a shame to have invested so much in an app that will immediately be shut down by CL.
Re:Search all Craigslist cities for a remote job by nydel on Monday April 08, 2019 @07:02PM (#19609975)
i'm not seeing this, has it already been cleaned up, or am i missing it?
Re:Search all Craigslist cities for a remote job by guessmyname on Monday April 08, 2019 @07:29PM (#19610186)
I clicked on the “search” button.

Then it sent 503 XHR requests:

https://i.imgur.com/R9D23iU.png

Re:Search all Craigslist cities for a remote job by jolmg on Monday April 08, 2019 @07:14PM (#19610085)
I see it.
Re:Search all Craigslist cities for a remote job by chovy on Monday April 08, 2019 @08:06PM (#19610405)
I know. It’s just a poc
Re:Search all Craigslist cities for a remote job by smt88 on Monday April 08, 2019 @09:08PM (#19610724)
Even a PoC needs basic security before being released publicly. It wouldn't even delay the release by a single day.
Re:Search all Craigslist cities for a remote job by everdev on Monday April 08, 2019 @09:22PM (#19610808)
Proof of Concept is a vague term and doesn't give you any sort of protection or defense. If it's against Craigslist's terms you're taking on some risk by making and public and promoting it.
Re:Search all Craigslist cities for a remote job by point78 on Monday April 08, 2019 @07:57PM (#19610355)
Craigslist will send you a take down notice in a week, this is explicitly against their terms. I'd advise taking it down before a lawsuit.
Re:Search all Craigslist cities for a remote job by ravenstine on Monday April 08, 2019 @08:54PM (#19610640)
That's basically what happened to Padmapper, sadly.

With the amount of scams causing a decline of their platform, I'd think that Craigslist would want other sources of traffic. I know that most sites have a policy against scraping/indexing, but Craigslist has their head up their own ass.

Re:Search all Craigslist cities for a remote job by rdiddly on Tuesday April 09, 2019 @03:01AM (#19612434)
I feel like anybody who quibbles about scraping has their head up there. It's like, are you on the internet or not? Yes? Then you are on the internet.
Re:Search all Craigslist cities for a remote job by biztos on Tuesday April 09, 2019 @05:38AM (#19613047)
Maybe timeliness is the important thing to Craigslist.

If you scrape all the jobs data as soon as it shows up, and repost it to your site, then maybe employers think you're the one they should be paying, based on prospects coming in your site and not CL.

I would assume that matters more than if you scraped older data or got older data by scraping it more slowly.

Re:Search all Craigslist cities for a remote job by tomca32 on Monday April 08, 2019 @07:03PM (#19609985)
Cool idea, but the implementation is kinda crazy. Good enough to just test the idea at home on your own machine, but pretty risky to expose it to the world and let a ton of other people go nuts making bazzilions of requests.

Also, exposed API key in the request.

Re:Search all Craigslist cities for a remote job by femto113 on Monday April 08, 2019 @09:13PM (#19610756)
I'm kinda curious how telecommuting to this job actually works. https://detroit.craigslist.org/okl/lab/d/asphalt-laborers/68...
Re:Search all Craigslist cities for a remote job by rxhernandez on Tuesday April 09, 2019 @12:49AM (#19611884)
Robots probably.
Re:Search all Craigslist cities for a remote job by whalesalad on Monday April 08, 2019 @06:51PM (#19609895)
Looks like we're performing a DDoS against https://api.scraperapi.com (the underlying data source) at this point.
Re:Search all Craigslist cities for a remote job by rkho on Tuesday April 09, 2019 @01:27AM (#19612036)
You should really consider caching results and only fetching them again after a certain amount of time has passed since the last time you scraped them. This way you're not blowing through your ScraperAPI limit.
Re:Search all Craigslist cities for a remote job by OrgNet on Monday April 08, 2019 @07:14PM (#19610084)
Play with dog remotely? https://i.imgur.com/OebiBWr.png
Re:Search all Craigslist cities for a remote job by giancarlostoro on Tuesday April 09, 2019 @07:41AM (#19613606)
Can you stack up on those and make 5 grand a month?
Re:Search all Craigslist cities for a remote job by hardlianotion on Monday April 08, 2019 @07:33PM (#19610209)
It is a very popular request.
Re:Search all Craigslist cities for a remote job by yubiox on Monday April 08, 2019 @06:26PM (#19609689)
Might want to take this out:

console.log(items);

Re:Search all Craigslist cities for a remote job by giancarlostoro on Tuesday April 09, 2019 @07:40AM (#19613604)
I usually set those to 'debug' instead, so I can filter them back into my JS console:

console.debug(items);

Re:Search all Craigslist cities for a remote job by dhritzkiv on Monday April 08, 2019 @06:26PM (#19609696)
The API it's calling seems to be overwhelmed at the time of this comment. The console is full of 429 (too many request) errors
Re:Search all Craigslist cities for a remote job by chovy on Monday April 08, 2019 @09:16PM (#19610774)
should be good now. i upgraded my plan.
Re:Search all Craigslist cities for a remote job by worg on Monday April 08, 2019 @06:28PM (#19609705)
It'd be nice to have a loading indicator.
Re:Search all Craigslist cities for a remote job by x2f10 on Monday April 08, 2019 @06:46PM (#19609853)
Thank you for this comment. I considered the webpage dead.
Re:Search all Craigslist cities for a remote job by noitsnot on Monday April 08, 2019 @10:32PM (#19611227)
Wow, I stopped it at 400 requests. I'm not sure how the site hasn't been shut down at this point.
Re:Search all Craigslist cities for a remote job by timsayshey on Tuesday April 09, 2019 @11:12AM (#19615428)
This is an amazing project! Probably one of the most useful ways to actually find quality job listings. If only Craigslist would offer site-wide search natively.

Btw, you should open-source this on GitHub. You should also include a crypto/paypal donation option.

Re:Search all Craigslist cities for a remote job by mothsonasloth on Tuesday April 09, 2019 @07:34AM (#19613557)
Its cool but the main problem I am facing with searching for remote jobs. Is that most remote US jobs are not technically remote. They are for US citizens only, so I can't work on it from Europe.
Re:Search all Craigslist cities for a remote job by codesternews on Tuesday April 09, 2019 @06:28AM (#19613263)
Hmmm.

"You're sending requests a bit too fast! Please slow down your requests, or contact [email protected] with any questions."

Re:Search all Craigslist cities for a remote job by momofarm on Monday April 08, 2019 @08:49PM (#19610617)
what I see the freelance job is playing with dog..... should there be a playing with cat job lol
Re:Search all Craigslist cities for a remote job by chovy on Monday April 08, 2019 @09:16PM (#19610772)
thosse freelance dog walkers are huge spammers. I've already removed duplicate titles across cities and you still see a ton of them.
Re:Search all Craigslist cities for a remote job by skizm on Monday April 08, 2019 @11:18PM (#19611472)
Can you just search Google for “<term> remote site:craigslist.org” for similar results?
Re:Search all Craigslist cities for a remote job by chovy on Monday April 08, 2019 @05:41PM (#19609319)
Just enter your search phrase like "node.js" or "javascript"
Re:Search all Craigslist cities for a remote job by ada1981 on Tuesday April 09, 2019 @07:24AM (#19613512)
All you critics are missing the obvious blue ocean market in remote dog entertainment services that clearly is begging to be served.
Re:Search all Craigslist cities for a remote job by starpilot on Monday April 08, 2019 @06:50PM (#19609891)
nice it doesn't work.
Re:Search all Craigslist cities for a remote job by anonytrary on Monday April 08, 2019 @10:05PM (#19611073)
I, too, entered a search query, pressed go, and witnessed absolutely nothing happen. Turns out, you just have to wait a bit. It scrapes Craigslist in real time from the client, so you have to wait a bit for results to pop up. It'd be nice if they had a loading/progress indicator.
Re:Search all Craigslist cities for a remote job by chovy on Monday April 08, 2019 @11:56PM (#19611658)
I'm working on an indicator next.
Re:Search all Craigslist cities for a remote job by point78 on Tuesday April 09, 2019 @03:01AM (#19612431)
Fill us in on your thoughts for not believing you will get sued by craigslist after multiple comments in the thread.
Re:Search all Craigslist cities for a remote job by sithlord on Tuesday April 09, 2019 @08:07AM (#19613754)
would this not fall into a similar category of happened to linkedin? The data is publicly facing, and not behind a paywall or a password, therefore, basically fair-use of internet?

ianal, nor do I know much about the subject, but seems like if its not at a minimum password protected information, it is pretty fair-use.

Re:Search all Craigslist cities for a remote job by fidla on Tuesday April 09, 2019 @07:33AM (#19613553)
hmm...doesn't work in chrome
Re:Search all Craigslist cities for a remote job by diminoten on Monday April 08, 2019 @07:07PM (#19610040)
Spamming 429s in console, oops!
Re:Search all Craigslist cities for a remote job by okay3637 on Tuesday April 09, 2019 @04:16PM (#19618515)
i need a job