This is awesome and much needed.

Just throwing it out there, but would you consider making a dump of the data you scraped that could be used by data scientists? Maybe as a torrent or something like that? Data about movies and what people say about them could form the basis of a lot of NLP projects.

What other big datasets are there for forum post text data? The reddit dataset most immediately comes to mind, and I've also seen a similar one for HN comments. Any others?

ArchiveTeam's web archives will be available to everyone without restrictions or profit as usual.

Well, thanks to your comment, I just found out ArchiveTeam exists.

Thank you!

Where you can download the boards?

Yeah that's definitely something I could do. Are there any proposed projects you know of that could benefit from this type of data?

Everything from simple sentiment analysis, to archive.org, to another mirror. I hope that does not discourage you from releasing the data.

Edit: I see the other comment about archive team already collecting and releasing this data, for free in an open format. I think that will be a good first source as well.

I think the priority was to reenable discussion of new movies and TV shows, that would be useful moving forward. But maybe they could make an API.

I'd love to explore how this data could be used to enhance recommender systems.

AFAIK, Jinni is doing exactly that.

Thank you for this. I'm curious if IMDB considers their (soon to be former) forum posts under copyright, or if you already have a plan in place.

One actually hopes that IMDB doesn't in fact have any claim on posts written by users that happened to be hosted on their site.

From http://www.imdb.com/board/announcement

"...disable our message boards on February 3, 2017 but will leave them open for two additional weeks so that users will have ample time to archive any message board content they'd like to keep for personal use."

Note the "personal use" above. Is a website like MovieChat.org considered personal use?

From http://www.imdb.com/conditions , section Copyright: "The compilation of all content included in or made available through any IMDb Service is the exclusive property of IMDb...".

Yeah that's a good question. Before I became an engineer I practiced IP/copyright law. Also spoke (colloquially) with a few ppl at the company.

I did my homework before starting this project and I'm confident we're in a good position :)

Could you explain why you think you're in a good position? From what I understand, the comments are owned by the users, and by posting them to IMDb, they allowed IMDb and their affiliates the right to use them.

The users did not give you permission to use their data, and you're not associated with IMDb, and therefore do not inherit the rights.

Lastly, IMDb specifically states that scraping or extracting data is forbidden without written consent.

>>Before I became an engineer I practiced IP/copyright law.

I'm sure there's a story there since I would have thought the usual direction, even for a small aggregate number of eventual engineer-patent attorney pairings, would be reversed. Hopefully you can post an "About Us" on MovieChat.org at some point when you feel comfortable. Thanks!

Interesting project? The IMDb message boards were a good way to get some additional insight after watching a movie. Or at least give me a good laugh at the trolls. :)

Who owns this post data? IMDb? The users? Do you have any plans or ideas for existing IMDb users to somehow claim their username on MovieChat, linking to their archived posts?

> The IMDb message boards were a good way to get some additional insight after watching a movie. Or at least give me a good laugh at the trolls. :)

Totally agree. I often also went there before watching a TV show or movie to get an idea what it was about or if I'd like it (though wary of spoiler potential), whereas I almost never read the reviews there, which I find are often far more useless than the message board comments.

After they're gone IMDB will be just a soulless movie listing site, with no reason to visit other than to check a mark out of ten, and no reason to stick around afterwards. :/

Hi thanks for posting, looks like it has some potential.

one item i didnt like is that if i search for a show, i get hits for each episode.

would be good to have one entry per show, and then you can drill down into individual episodes.

also would be good feature on ability to rate shows (as a registered user, but i guess IMDB is still offering that functionality)

So basically reimplement IMDB?

I made a userscript to be a drop-in replacement for the current discussion-teaser on a movies page on imdb, but it gets the discussions from moviechat.org (and links to them directly).

For it to work I would need for Jim to enable Access-Control-Allow-Origin from imdb.com (and ideally make a json/xml version of the thread list since I'm regexp'ing the html at this point).

Maybe it would help persuade him if we showed that there was some interest for this.

https://imma.gr/66880x4ca8b

Hey Jim – thanks for doing that! I'm curious, how did you back up IMDB's message boards? Did you scrape the whole website or were you able to find a data dump?

I wanted to ask this too. Did you scrape it or did it via some api? Anyway, cool project, wish you all the best with it.

illegally scraped,IMDb doesn't have API for this and automated scraping OP done is against their TOS, not mentioning without consent of users, for instance stole 14 years worth of my posts

He stole them? You mean they are gone now? Because the closest I can imagine happening is an unlicensed reproduction of your content, your credentials included. ... Like any archive service would do it.

I take it then, when you posted these 14 years of comments, you didn't want them available to the public to, I don't know, read?! Thank goodness IMDB deleted them all before we happened across them by accident & violated your privacy.