News for hackers. Stuff that matters.
Re:Craigslist posts shown on a map while viewing Craigslist ... by chimi on Wednesday August 01, 2012 @03:43PM (#4325182)
In the end, client side scraping of the data actually increases the traffic on CL's servers, because multiple people have to scrape the same posts, rather than a central server. If padmapper remained small, then they probably scrape more posts than they have unique views. Of the 100 they scrape, maybe only 10 are seen. As padmapper et al grow, it may reduce the bandwidth on cl.

In 10 years, if all clients looking for apartments use a plug-in however, now you have cl paying for a lot more bandwidth than outsourcing this functionality to a centralized service, because for every 100 posts they scrape, they may only really look at one and they want the ability to show thousands on a single map.

I really don't understand why cl doesn't just do it themselves. It's not hard. I've built these maps countless times on top of all the popular mapping services and using different geocoders.

The hardest part is figuring out what the poster means in their location: where is "forth and main" [sic] or do you mean "East 4th St at Main St?", but I can see a simple address validator that updates a map with their location until they get something that will geocode correctly. That's the best solution and everyone wins here, except padmapper and the like, but there are other sources of data to map that aren't from craigslist too.

It's really trivial. The only other issue is cost, but those can be covered easily with all the money CL leaves on the table.

Re:Craigslist posts shown on a map while viewing Craigslist ... by olog-hai on Wednesday August 01, 2012 @04:23PM (#4325425)
> client side scraping of the data actually increases the traffic on CL's servers

Of course, but not in PadMapper's case.

"Padmapper gets its data from 3Taps, a site that collects and republishes Craigslist's content. 3Taps, in turn, says it gets its data from cached copies supplied by Google and Bing, rather than Craigslist directly."

Source: http://arstechnica.com/tech-policy/2012/08/craigslist-tighte...

Re:Craigslist posts shown on a map while viewing Craigslist ... by cdibona on Wednesday August 01, 2012 @05:16PM (#4325785)
I'm sorry, but you should really ask yourself if this is true. Bing maybe, as they can do what ddg does and pull a feed from Yahoo/Bing BOSS (maybe?) but to scrape Google would be a very bad and unreliable (and counter to the Google TOS) way of getting this information.

This whole situation is very sketchy, honestly.

Re:Craigslist posts shown on a map while viewing Craigslist ... by cbr on Wednesday August 01, 2012 @05:37PM (#4325891)
Padmapper used to scrape Craigslist, but stopped after a cease and desist letter. While buying the data from a service that scrapes search engine caches may be unreliable, my understanding is that it's legally much more defensible.
Re:Craigslist posts shown on a map while viewing Craigslist ... by jeremyarussell on Wednesday August 01, 2012 @04:37PM (#4325529)
Is that argument in consideration of the fact that padmapper started off by scraping CL directly and forwarding everyone back over to CL?
Re:Craigslist posts shown on a map while viewing Craigslist ... by jack-r-abbit on Wednesday August 01, 2012 @05:25PM (#4325832)
They used to do that. They were told to stop. They stopped. They found a new way that didn't affect CL bandwidth. So what they started off doing is not all that relevant to what they are doing now. However, what this Chrome Extension is doing is pretty much what PadMapper was doing... so it would follow that this Chrome Extension will get smacked. Or worse... the extension users each get smacked.
Re:Craigslist posts shown on a map while viewing Craigslist ... by jack-r-abbit on Wednesday August 01, 2012 @04:01PM (#4325292)
Not only that, but for every housing page with a list on it, the scraper is going to pull all those pages to parse the data and display on a map. I suspect a person who is doing a lot of apartment hunting with this extension runs the risk of just getting their IP address banned. I assume CL has that capability.
Re:Craigslist posts shown on a map while viewing Craigslist ... by welder on Wednesday August 01, 2012 @04:11PM (#4325349)
Since users will be browsing the CL website, CL will still control the distribution of their content. I'm betting tech details are just details as long as users are browsing the CL website.
Re:Craigslist posts shown on a map while viewing Craigslist ... by joe_the_user on Wednesday August 01, 2012 @06:20PM (#4326089)
Yes,

I don't CL ever made the argument that the problem was traffic. The whole "we're not hurting your servers" thing seems like a strawman to justify violating CL's TOS.

Re:Craigslist posts shown on a map while viewing Craigslist ... by aidenn0 on Wednesday August 01, 2012 @06:23PM (#4326099)
CL repeatedly claimed that traffic was a major issue with scraping.
Re:Craigslist posts shown on a map while viewing Craigslist ... by pavel_lishin on Wednesday August 01, 2012 @10:48PM (#4326949)
> I really don't understand why cl doesn't just do it themselves. It's not hard.

It's not hard to do a mediocre job, but it is super hard to build something really great.

And it's super easy to build something so shitty, that it'll drive away the users that you have now.

Re:Craigslist posts shown on a map while viewing Craigslist ... by loumf on Wednesday August 01, 2012 @04:02PM (#4325297)
I think CL is worried that their bandwidth would start going down as people start putting their apartment listings in PadMapper instead of CL because CL doesn't have any apartment listings any more.