News for hackers. Stuff that matters.
Re:I Want to Fix Goodreads by johnfn on Saturday September 12, 2020 @03:57PM (#24455077)
Good lord what I wouldn’t give for a better goodreads. Never has so much useful and actionable data been squandered. For example, I could write pages about how bad the “top books” lists are. The Goodreads Choice Awards are purely and literally popularity contests, for example. Why would this be the case - you have millions of user ratings, you should be using those to surface exciting and unknown books rather than throwing people back at the same few authors they already know. The lists work exactly the same way - they discard ratings entirely and only take into account the quantity of books. It would be as if you said that the top 40 is the best music because everyone listens to it.

So you don’t think I’m all talk, a while back I got so fed up with this that I wrote my own script to scrape goodreads and find the actual best books, not just the most popular, and I found a wealth of really good and unknown books, including two books that are now my favorite books of all time. It was a side project that took me an hour. Why goodreads can’t do this is utterly beyond me.

Aaaargh!!!

Re:I Want to Fix Goodreads by taeric on Saturday September 12, 2020 @04:05PM (#24455159)
I'm confused. What you describe is really just another form of popularity contest, no? Unless you have a set of reviewers that you value over others. One might call them the critical reviewers. (No, I'm not trying to be subtle)

So, what books did you find? Why not make a post on your method?

Re:I Want to Fix Goodreads by johnfn on Saturday September 12, 2020 @05:29PM (#24455926)
Not exactly. Goodreads will value a million people giving a book 1 star higher than 20k giving it 5 stars. I just sorted by average score.

I posted the books in this thread (sorry, on mobile right now).

Re:I Want to Fix Goodreads by taeric on Saturday September 12, 2020 @06:20PM (#24456376)
But that is still a popularity contest, of sorts. In general, any ranking system is a popularity system. Which is logical, as there is no objective ranking of books. That is, it isn't an ordered set.

That said, I can see value in providing a score not just in the items, but on the ranking system used. Could even be adaptive based on genre. (Indeed, I would love a system to see if any books have factual errors and such. But, that is just trying to reinvent the citation systems of academia.)

And understood on the mobile. Same boat for me.

Re:I Want to Fix Goodreads by johnfn on Sunday September 13, 2020 @11:18AM (#24461394)
I don’t understand. A book with a single review could place higher than a book with millions. How is that a popularity contest?
Re:I Want to Fix Goodreads by taeric on Sunday September 13, 2020 @02:07PM (#24462712)
Depends on who reviewed it.

Easier to think of this from the other side. I have a few friends that, if they tell me to avoid something, it can typically override however popular something may be.

Re:I Want to Fix Goodreads by xuhu on Saturday September 12, 2020 @04:11PM (#24455211)
How did you define the actual "best books" ? I usually pick a group (Goodreads users in country x), scrape the books of all its users, and sort them by frequency: https://github.com/harjoc/goodreads-group-books
Re:I Want to Fix Goodreads by bbarnett on Sunday September 13, 2020 @04:30AM (#24459362)
Thinking on johnfn's response, and yours, these are all simple SQL queries. You could easily delineate to others what each query does, and let people choose from a list.

EG "best average" "most reviewed" "users in country + frequency" and such.

This doesn't even need to hit performance, as these queries could all run on a secondary SQL server, be cached in redis/whatever once an hour, or even day.

After all, this sort of list output doesn't need to be second by second changing.

So simple to implement, non-performance impacting, which makes it even more curious as to 'why not'. Likely, a UI/UX person thinks it will 'confuse' people.

(I know that may not be the case, but it seems to kill more sites, and more flexible functionality, than anything...)

Re:I Want to Fix Goodreads by PaulStatezny on Saturday September 12, 2020 @03:59PM (#24455111)
Would you share the titles and authors of those two books?
Re:I Want to Fix Goodreads by johnfn on Saturday September 12, 2020 @04:40PM (#24455452)
Infinite jest by David foster wallace and only forwards - forget the author name off the top of my head. Infinite jest is pretty popular these days. The other I have yet to hear anyone say they have read.
Re:I Want to Fix Goodreads by SamBam on Saturday September 12, 2020 @09:23PM (#24457391)
Ok, but was Infinite Jest ever an "unknown" book? I'm pretty sure it was one of the most hyped books in the world the year it was written. And I'm pretty sure it's always been in the 4.x's on Goodreads.
Re:I Want to Fix Goodreads by johnfn on Sunday September 13, 2020 @11:20AM (#24461409)
I’d never heard of it nor seen it on goodreads till I made my list. I’m sure it’s popular in some circles, but not mine.
Re:I Want to Fix Goodreads by defen on Saturday September 12, 2020 @09:07PM (#24457333)
Liking Infinite Jest suggests an interest in American literary maximalism and / or comic novels.

Suggestions: The Sot-Weed Factor, Gravity’s Rainbow, Mason & Dixon, Underworld, and if you’re feeling particularly masochistic The Tunnel

Re:I Want to Fix Goodreads by tomstuart on Saturday September 12, 2020 @05:38PM (#24456027)
I’ve read Only Forward by Michael Marshall Smith. It stuck in my mind the same way that Vurt did. I still think about the man with no head sometimes, and the idea of a road being different when you’re going the other way on it.
Re:I Want to Fix Goodreads by jordiburgos on Monday September 14, 2020 @05:53AM (#24468420)
Please share the results of your script !! :)
Re:I Want to Fix Goodreads by johnfn on Monday September 14, 2020 @06:34PM (#24475911)
Posted it elsewhere in the thread - go check it out.
Re:I Want to Fix Goodreads by iamkroot on Sunday September 13, 2020 @12:48PM (#24462094)
Can you share this script?
Re:I Want to Fix Goodreads by johnfn on Monday September 14, 2020 @06:33PM (#24475909)
Wow, turns out I still have it on GitHub, including the results I ground out 8 years ago. The script probably doesn't work, but the results are still good:

https://github.com/johnfn/GoodReadsScraper

Particularly, load bigdata.js into nodejs and then run a command sort of like this to parse out the results, filtering out young adult/romance/religion stuff/comics:

    bigdata.filter(f => f.ratings > 5000 && !f.genres.includes('Young Adult') && !f.genres.includes('Religion') && !f.genres.includes('Romance') && !f.genres.includes('Sequential Art')).map(f => f.title)
I get some pretty interesting stuff. First result is the Constitution... OK, fair enough... but the next 10 or so are:

    'A Song of Ice and Fire',
    'Collected Fictions', (by Borges)
    'The Name of the Wind',
    'Infinite Jest',
    'The Complete Works',
    'The Way of Kings',
    'The Wise Man\'s Fear',
    'Ficciones',
    'A Storm of Swords: Blood and Gold',
    'The Complete Stories',
    'Labyrinths',
    'Don\'t Let the Pigeon Drive the Bus!',
    'The Hiding Place',
Is it an amazing list? Eh. Is Name of the Wind better than Infinite Jest? Probably not. They're both fantastic books, though! And still, it's way better than Listopia. Also, this data is 8 years old. I bet it would be way better if I were to clean it up and run it in 2020.