I'm most interested in this piece of the project. What were your particular tools and methodologies? How long did it take you, once you identified your data sources? Any interesting stumbling blocks or problems that were solved along the way?

Hot tips for crawling data:

  - Cache pages locally while you work on the indexing
  - Nokogiri is awesome
  - Don't be afraid to use regular expressions
  - Initially, put data into a spreadsheet (not the db).
    That way it can be checked in and diffed.

  # replace all vowels with the letter E
  s = s.gsub(/[AEIOUY]+/, "E")

(edit: formatting)

1. Caching pages is definitely a great idea while debugging. Especially if the data source has a request limit :)

2. I've never heard of Nokogiri, but it looks like BeautifulSoup for Ruby. I've found that Python has worked for everything I need so far, but thanks for the reference.

3. I suck so bad at regex, but using it more will help me climb that mountain.

4. One tip I've used is writing out the "INSERT INTO TABLE..." statements along with the scraped results. I definitely use CSV (and Google Refine) for general clean up and spot checking.

5. You should write a 'Data Scraping One-liners Explained' ebook :)

Then you are the opposite of where I used to be -- I thought I understood and could use regexps. Hell, I do Perl for fun. :-)

Something like when I first sat down with Photoshop -- "Hey, I know how to program Macs [before MacOs X]. This is just using a Macintosh program, so I should have no problems"... :-)

Read "Mastering Regular Expressions". It made me feel embarrassed about my previous stupidity [Edit: Embarrassment, your name is Dunning–Kruger :-) ]. Just the first few chapters are enough to change your world.

Edit: I might add, I still can't use Photoshop.

For tricky pages, it can be a good idea to write tests though. Be sure to cache all downloaded pages so the tests can be run uber-fast.. It's a good way to perfect that regex. (Yes, there are programs for that, but sometime HTML have some weird newlines or characters that screw everything).

Also, I think it's important to emphasis not to "code everything". Say you've got 10 links to get on the first page.. and in all of these links you've got hundreds. Obviously, you won't go on each page and manually copy everything; however, for the first 10 links, it's useless to write a script that crawl that. Just copy it and clean it from the source. Jee, use a macro in emacs or your favorite editor if you really cant repeat a task 10 times.

Suggests:

* show me excerpts on-page (if possible from amzn?)

* allow community +1/-1 on books and generate lists based on top-rated by site users

* commenting, facebook or disquss, on each book

(little issue: Facebook "like" button on main page and book pages doesn't seem to be working -- dunno if that's facebook's problem not yours)

I built it on Rails as well

http://www.geekwire.com/2011/hunger-science-fiction-books-sp...

Also, thanks to webwright for the title suggestion.

I'm sort of slowly working my way through the double winners of the Hugo and Nebula (kind of a lifetime goal, I guess).

It took me a while to find the "Hall of Fame" for books, but that is what immediately wanted from a site like this (I don't care so much about the year-by-year rankings). Maybe make it more prominent?

EDIT: also, I think there's a big difference between nominations and winning. Would be cool to sort based on actually won awards, not just nominations.

Also I have a series to add: Harry Turtledove's "Darkness" series. It is like 5 books, if I remember correctly. I'm at work...

http://www.bestsfbooks.com/b/3119/The-Children-of-the-Sky

The Children of the Sky Series: Zones of Thought #3 by Vernor Vinge (Tor, Oct-2011)

Only glitch I've run into so far is that the search function doesn't work. As far as I can tell, there's no type="submit" for the form.

Maybe it would be an idea to show a message like "No results..." if what the user types does not return any matches in this search function?