curio

thanks for taking the time to respond to this. i definitely understand where you're coming from and am glad to hear the robots.txt is being updated.

but their robots.txt says that you're welcome to scrape it:

  User-agent: *
  Disallow: /ac.php
  Disallow: /ae.php
  Disallow: /album.php
  Disallow: /ap.php
  Disallow: /feeds/
  Disallow: /o.php
  Disallow: /p.php
  Disallow: /photo_comments.php
  Disallow: /photo_search.php
  Disallow: /photos.php

incredibly easy.

and not only can the bot lie, it can disregard the robots.txt file altogether. just like the terms of service document for humans, you can choose to disregard it & deal w/ the consequences (blocked IP's, lawsuit, etc).

robots.txt is just a version of the TOS that computers can read.

yes, using the user-agent:

http://en.wikipedia.org/wiki/Robots_exclusion_standard

this is a really important issue.

this new model that facebook is trying to push isn't scalable. it favors the big guys and it's bad for the open web.

you shouldn't have a TOS that contradicts your robots.txt. period.

probably not, but that's irrelevant. the point is facebook's robots.txt says you can crawl it but their TOS says you can't. facebook have already shown they are willing to sue anyone who tries to crawl without permission. this new model that facebook is trying to use isn't scalable, it favors the big guys and is bad for the open web.

you can also do this with a regular expression. here's the blog post i wrote on it a year ago. i use one function to grab get variables from either window.location or from a script tag. 5 lines of code.

http://www.onlineaspect.com/2009/06/10/reading-get-variables...

and just like chatroulette or most anonymous forums, the tone quickly deteriorates to the level of graffiti in a public bathroom stall.

he should drop out of school and use it as an excuse to get on every tv station and written about in every paper in the country.

who wants a degree from such a dumb school anyway. walk away. some of the best companies of our time have been built by drop-outs.

practically speaking, most respectable email services will have an MX record. i tested it on about 30,000 emails & a simple MX lookup caught about 40 gmial's and only 1 false positive.

that reminds me of the girl who takes advantage of the zappos free returns policy and gets 5 new pairs of shoes every singles week only to mail them back a few days later. but zappos doesn't cut here off -- she is important evidence that they are serious about their policy & for zappos there are enough honest people out there to more than make up for her.

and while we're on the topic of how broken SSN's are...

why is it that the most important piece of identification you receive in your life, that you also need to keep forever, is printed on a crappy piece of paper???

It sucks that comments aren't enabled on that post. You can see the responses from the community (including Brett Slatkin one of the authors) on the Google Group for PuSH:

http://groups.google.com/group/pubsubhubbub/browse_thread/th...

YES!

That's an important point. You can see how PuSH handles it here: http://pubsubhubbub.googlecode.com/svn/trunk/pubsubhubbub-co...

Crap - don't know how that happened. I've reposted the article with the correct link if you care to thumb that one up instead.

It should be back up now. Too much traffic I guess. :)

Impressive demo. Very cool to see real-time updates coming to Google Reader.

I don't know what you should do, and I don't necessarily think there is one right answer for everything in this position. I do know that almost every successful entrepreneurship I've talked to has a story about a failed relationships because of the sacrifices they made for their company. You've gotta weigh the costs for yourself and decide what you want most. This question reminded me of this article by Steve Blank. Hopefully it will help.

http://steveblank.com/2009/06/15/lies-entrepreneurs-tell-the...

+1 vote for pubsubhubbub

must have 500k in annual sales and 51% ownership to apply!

I recommend Powweb to everyone. Reasons why I love them:

- cheap: $7.77/month

- reliable: i've never seen them go down

- advanced developer features: cron, .htaccess, php.ini editing

- really clean user interface (forget using godaddy)

- amazing customer support: call them. they answer first ring. there's no waiting for hours after trying to navigate crappy phone tree systems.

i've used a dozen different shared web hosts over the years and the vast majority of them suck. Powweb really stands apart from the crowd, especially when it comes to advanced features and customer support. When I was doing web design consulting I moved every single one of my clients over to Powweb -- they're that much better.

http://www.powweb.com/join/index.bml?AffID=563885