Re:Good News: Craigslist drops exclusive license to your posts
by dredmorbius on Thursday August 09, 2012 @06:31PM
(#4363667)
Last I checked, robots.txt excludes all crawlers from at least several major classifications, including housing listings (/hhh):
User-agent: *
Disallow: /cgi-bin
Disallow: /cgi-secure
Disallow: /forums
Disallow: /search
Disallow: /res/
Disallow: /post
Disallow: /email.friend
Disallow: /eaf
Disallow: /reply
Disallow: /?flagCode
Disallow: /ccc
Disallow: /hhh
Disallow: /sss
Disallow: /bbb
Disallow: /ggg
Disallow: /jjj
Disallow: /*rss$
http://www.craigslist.org/robots.txt'jjj' is jobs, 'ggg' is gigs, 'bbb' is services, 'sss' is for sale, 'ccc' is community, 'res' is resumes.
Re:Good News: Craigslist drops exclusive license to your posts
by danso on Thursday August 09, 2012 @07:02PM
(#4363813)
Right...but not apartment listings (aap)
http://newyork.craigslist.org/aap/ http://newyork.craigslist.org/roo/ http://newyork.craigslist.org/sub/
Re:Good News: Craigslist drops exclusive license to your posts
by dredmorbius on Thursday August 09, 2012 @07:20PM
(#4363882)
Point. I mistakenly assumed /aaa covered all of housing.
Looks like the same principle applies to a number of other subcategories. So while overview indices aren't spiderable, subindices are.
Also, CL has not removed itself from search engines. They're using NOARCHIVE.