IMDb forbids scrapers in their conditions[1] but you can freely download their datasets[2].
Similar to what you described: omdbapi[3] is a third party API for the free IMDb data.
[1] https://www.imdb.com/conditions
https://www.eff.org/deeplinks/2018/04/dc-court-accessing-pub...
suggests there is little to no recourse for IMDB and the likes. Craigslist was able to win their case against 3taps, arguing the scraping was putting a load on their servers (typical Craig Newman bullshit) and that they continued scraping even after the IP ban and that is a computer frauds act or something like that which is draconian response likes of which that guy who killed himself because he got caught for scraping academic journals.
IIRC Pushshift is what powers sites like https://removeddit.com/
(to anyone reading this, please consider donating--this project is so cool)
1. I did something very similar and got a legal threat from a Fortune 100. My attorneys had successfully litigated similar matters but told me in my case, the law was clear-cut in the F100's favor. My attorneys attempted to negotiate a license agreement that would allow me to continue, but it fell on deaf ears and I had to shut down my business.
2. Like most things on the internet, sure, you could. But someone could stop you with legal force if they were inclined to do so.
Even though cases like Feist supposedly say that facts pulled from fact-collections aren't subject to copyright, that doesn't really work out on the internet for a couple of reasons.
The biggest reason is the CFAA. Anyone has the right to tell you to stop talking to their servers at any time. Such notice doesn't even have to be explicit; in Craigslist v. 3Taps, the judge indicated that IP blocks should've been sufficient notice that 3Taps's presence was unwelcome, but companies frequently go even farther and say that the fine print buried in the Terms of Service banning "automated access", "spidering", "crawling", or similar is sufficient notice. Judges frequently accept that argument.
Failure to heed these notices very likely constitutes "exceeding authorized access" under the CFAA, which is both a tort and a crime (meaning, you can both be sued and be put in jail for it). Companies use this to bludgeon upstarts all the time.
The other thing here is:
a) very few facts on the internet are really alone; interesting data is almost always encased in HTML or some other copyrightable superstructure.
b) a legal philosophy called "the RAM Copy Doctrine", which is very well established at this point, means that storing any copyrightable work in system memory for any amount of time, even if it's just long enough to extract a non-copyrightable fact and then throw the surrounding copyrightable content away, meets the legal definition of a copy and is thus potentially infringing. This effectively allows rightsholders to instantly double any infringement claim, and attorneys actually have tried to posit that each instance of a copyrighted work getting loaded into memory for e.g. network transmission to another source should count as an independent act of infringement.
The 9th Circuit ruled that memory copies meet the Copyright Act's definition for fixed, non-transitory copies in MAI v. PEAK (1993) and that has been (mis)applied to most digital copyright controversies since. This is the effective equivalent of stating that the reflection of an image from the human retina onto the optic nerve is a separate potentially-infringing copy.
Scraping is most certainly still a legally dubious business. You have to take the Google route and get too big to fail before you'll start to see inconsistent outcomes like Perfect 10 v. Amazon, where the judges effectively gave Google a pass because they were scared of the widespread ramifications of shutting down Image Search, despite the clear legal reasoning that would've mandated this.
IANAL
Here are a couple of cases that I'm especially interested in:
1. Does this mean that it is legal to scrape "database"-type websites for statistics and provide them on your own website? Could one use this ruling and copy all of IMDB's film data? Repackage that data into a Creative Commons website? Or a better set of tools for casting agents?
2. What about social or community-curated websites? Could you mirror all Reddit comments (which used to be Creative Commons anyway) to a more dev-friendly site? Don't force a mobile app down people's throats? Make it ad-free, donation-supported like Wikipedia?
3. What about big media? Could you bootstrap a new video site by scraping all existing (or popular) YouTube videos? Provide a means for owners to "claim ownership" of their account on the new site? Then market it as YouTube "but grown up" (18+)?
4. Kind of getting off-topic, but could you build a new music service by temporarily ignoring copyright, copying pirated music, then pivot to something that does collect money (via ads or subscriptions) for the music rights holders? I'm thinking Spotify, but built for music connoisseurs. Rich APIs with tagging, smart playlists, etc.
How likely would any of these be to avoid lawsuits until they're big enough to hire a legal team? Are certain behaviors less legal than others?
I'd really appreciate feedback on this. (Thanks in advance!)