Show HN: Curated, Pre-trained ML Models for Transfer Learning

Re:Show HN: Curated, Pre-trained ML Models for Transfer Lear... by minimaxir on Tuesday February 13, 2018 @02:39PM (#16370029)

Two of my projects, textgenrnn (https://github.com/minimaxir/textgenrnn) and reactionrnn (https://github.com/minimaxir/reactionrnn) are among the pretrained models.

While this is allowed by the MIT License (and there is sufficient attribution to the source repos), it might be helpful to more explicitly state that the curated models are forks/modifications. These models also have a dependency on the source packages, which I can't promise that it won't have breaking changes if I do decide to update the package.

I do like the new accompanying examples for those projects, and it's good to see the projects actually being used! :)

[ Parent | on HN ]

Re:Show HN: Curated, Pre-trained ML Models for Transfer Lear... by rm999 on Tuesday February 13, 2018 @03:44PM (#16370572)

Hey, I just installed and played around with reactionRNN; I hate to be negative (especially to people open sourcing models, kudos!), but your model seems to perform quite poorly. It immediately failed my "easy" smell tests: https://i.imgur.com/FvfuZgy.png, and didn't really work on most of my other tests: "This book sucks" is 0% angry, "I'm going to go home and listen to emo music and cry" is 0% sad, "Check out this hilarious youtube video" is only 26% haha. Your example "He was only 41." is 100% sad, but "He was only 42." is 0% sad. These aren't hand-picked, these are literally things I just typed in. From what I can tell it usually gets anything negative wrong, and usually picks "haha" for the positive ones.

Subjective performance is worse than your included examples. I've been building models my whole career, and what I've learned is most people will take claimed performance at face value until it burns them. It's beneficial to no one if someone comes up with an idea based off your repo description, builds it out, then finds it doesn't work adequately. My advice is to update your examples and test cases, and keep finding ways to improve the model.

[ Parent | on HN ]

Re:Show HN: Curated, Pre-trained ML Models for Transfer Lear... by minimaxir on Tuesday February 13, 2018 @03:54PM (#16370663)

The result is the reaction to a given text; it won't fully be the same as a sentiment analysis, unfortunately.

Additionally, as I put in the README notes, keep in mind that the network is trained on modern (2016-2017) language. As a result, inputting rhetorical/ironic statements will often yield love/wow responses and not sad/angry. That type of systemic bias in text analysis is currently unsolved and there isn't an easy way to account for it.

"I am so angry" and "I'm going to go home and listen to emo music and cry" are phrases that would likely be posted on Facebook ironically, and therefore classifying it tricky.

For the other examples, yes, that result might be overfitting on characters and while setting up the model I had difficulty accounting for that while still getting the model to converge.

I'll admit it's not a perfect model (it was a side project while I was frustrated during a job hunt), but it's a great proof of concept. Unfortunately, since Facebook crippled their /posts endpoint, I can't get more data to improve the model...

[ Parent | on HN ]

Re:Show HN: Curated, Pre-trained ML Models for Transfer Lear... by hsikka on Tuesday February 13, 2018 @03:09PM (#16370275)

Hey, thank you! That is an excellent point, and definitely a huge potential problem that we're gunning to fix, both in the submissions already up and those that will come in the future. With this version of ModelDepot, we really just wanted to pique people's curiosity and get them to see the power and potential of sharing well documented models.

Also, thank you for contributing to the ML community at large! I really enjoy your work personally, and hearing that you dig the examples we patched together means the world :)

[ Parent | on HN ]