Yes - it's a difficult problem. It must be possible to build a better statistical model of each rater, in order to weight their opinions. A first step would be to normalise the rating distribution of each person (e.g. by average and standard deviation). I wouldn't use a rating for a particular book, unless the rater had some minimum number of ratings, or a book had very few ratings.