There are other social book sites out there, and they seem to do well. In our case, the difference is that user intentions are clearly about dating. I think (hope) that matters.
So yeah, niche dating it is, which is why I suggest that you tone down the dating aspects of it.
And while it might be a small niche, this is a _huge_ market and attacking a niche then working out how to expand (movie matching? music matching?) is a perfectly valid strategy.
Basically, a site that is good for hookups or dating makes it very easy for users to quickly browse through the other users' images. The best ones I've seen basically lets you make a (detailed) search, then just shows the results one by one with a big NEXT button on each. No lists, no clicking back and forth, no popups, just a very simple flow through the results.
But matchmaking needs a workflow that is more refined and feels less like a flesh-market. In the market research we did where I worked many years ago we found out that matchmaking appeals to people with little time. They don't want to spend hours browsing profiles, they want to be fed a slow stream of new profiles that the system has decided is good for them. This could possibly work with book taste as a base. If you make a recommendation engine like Netflix or Amazon, you could then grab other users that actually liked the books and have that as a starter.
"User X shares your taste in books, but hasn't read book Y, why don't you tell him/her about it?"
Doing dating through book taste doesn't exactly signal hookups, so why even label it as a dating site? Make it a social site around book taste and let dating happen in the community by itself?