They index plenty of their content [1][2]. What they don't index is content not explicitly marked as "Public". e.g. Facebook posts with visibility settings or protected tweets. FB, Twitter, and now LinkedIn have plenty of content that's not publicly accessible: that is, content which doesn't require a logged-in account that explicitly agreed to TOS/EULA, but they have tons of publicly-accessible content, too. The latter is fair-game.
That said, the Craigslist lawsuit is still bewildering to me. That content is explicitly public, does not require a login, and the only agreements are the automatic unforceable browsewrap ones. The LinkedIn v. HiQ case is very similar to the Craigslist v. 3Taps, however the decisions are in opposite directions.
Published does not mean uncopyrightable. A radio station broadcasts a song; that doesn't remove the song's copyright protection. Receiving a copy is not making a copy.
3Taps settled put of court; there was no decision.
Padmapper is just a much better interface for craigslist data. Even though craigslist pulled their data, craigslist didn't create a new interface to match or even bother to improve their experience, even though they have the user mindshare for posting this type of data.
Essentially it's as you said - innovation is being killed to preserve walled gardens like Craigslist, and users are the ones being hurt here.
That's not completely true. After lots of bashing online [1], craigslist started adding some features such as searches on maps[2]. Even though it's not as nice and easy to use as padmapper was, it would still be considered an improvement.
[1] - https://www.inc.com/abigail-tracy/craigslist-quietly-makes-c...
[2] - https://sfbay.craigslist.org/d/apts-housing-for-rent/search/...
Example: Does it matter whether the data is ever stored? Say I make a listing notifier that pings people when a house is listed in their desired area. I’d need to crawl real estate listings and but I’d never store or republish any content from the site.
It's actually that I "read" the data, regardless of my intention and regardless of what I intend to do with it. But taken to the extreme there has to be something that makes it legal for me as a consumer to scrape ONE item from their data, while at the same time making it illegal for a company to scrape ALL their data in this way.
The worst situation to be in is one where it's "legal until it isn't" i.e., when you actually cause a problem (bandwidth, revenue) you will be sued - and until then you no idea.
Technically (and it's complicated these days because copyright law was mostly written well before computers and the internet existed) any time you create a copy of someone else's creative work, you need an explicitly granted right to do so. You can _read_ a book, but you cannot write down all or part of what you just read. You can listen to a song, but you cannot perform all or part of that song (An Australian band called Men At Work lost a copyright case for a flute solo that contained 2 bars/11 notes of a 1930's folk tune "Kookaburra" who's copyright had been sold/bought in 2000.)
There are some explicit legal exceptions to requiring a license, most commonly people talk about "fair use", but that's again a "pre internet" body of law, and is extremely open to interpretation as to how it applies to digital copies in computers.
https://www.copyright.gov/fair-use/more-info.html
https://fairuse.stanford.edu/overview/fair-use/what-is-fair-...
https://www.eff.org/deeplinks/2018/04/dc-court-accessing-pub...
It's foolish to assume that the lawyers and the justice system will help you out. This is simply not a case of somebody pirating a movie and slapping on their own copyright.
CFAA was how craigslist won but that's been thrown out thanks to the EFF lawyers.
Going forward, there will be little to no recourse for "web scraping public data".
Suppose there was an intermediary platform so that the user uploads his content/video once to the intermediary platform, and can then select which platforms may use the content under what conditions (i.e. perhaps for free, perhaps for a user-set minimal amount of remuneration, ...), or perhaps instead of selecting which platforms, selecting the rules to which a platform must adhere in order to distribute the content (so that new players can instantly start hosting a diverse set of content as long as they adhere to these rules). The user thus only needs to upload once.
Obviously it would require youtube etc, to provide API access for the intermediary so it can post the content without being able to hijack a user's account on youtube etc...
This would not be in their interest of course, so anyone attempting to create this intermediary upload service would face a never-ending cat and mouse game, somewhat reminescent of youtube-dl...
Perhaps instead of silly link tax rules, it would be better for governments to force platforms above a certain size/usage to provide API access such that these intermediary upload services can flourish.
They might effectively become some kind of consumer/producer protection organizations, by proposing multiple usage standards (like my video MUST NOT have ads, or perhaps the opposite for another user, my video MUST be remunerated at x cents per y views, otherwise the platform may no longer distribute my content, ...) from which their users can select...
apart from "pure format" content like music, videos, ... as soon as there is more compound structured content, like a blog post with images, it would be hard to standardize the format.
then there is the problem of authentication, especially if a user uploads a video through the intermediary, and 1 year later a new platform wishes to distribute the content on its network too, and after another year the user wants be active on this platform, there is already an account, and how does he get his login credentials?
In theory the intermediary could have a messagebox for each user, and the platform can send the credentials to the user in this messagebox, so that if interested a producer can decide to make use of his account on a new platform...
Edit: so while currently all the platforms act like publisher and printer separate the market into publishers (intermediary rights managers) and printers/distributors (big platforms like YT etc). Similar to how humans decided it was better to split up doctors and pharmacists to prevent de facto quackery of selling whatever you happen to have on hand, or cheaper deals for...
My company, Datastreamer, has been in business for ten years indexing public web content (news + blogs). We focus primarily on "live" content. Content that publishes often.
The main challenge we've always had is that just because the content CAN be indexed doesn't necessarily mean you MAY index it.
A recent situation was around Craigslist vs 3taps:
https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc.
Basically the issue doesn't evolve around WHO has copyright but who has copyaccess to the content.
So if you create an account on Acme.com... You still own the copyright to the content you post but Acme controls access. Not only that but the ToS that you sign gives them rights to your content including bulk sales.
This means that Acme can monetize the content that YOU create while actively preventing people from indexing it even that may be your intention.
This means that in 2018 a company like Google COULD NOT get started because websites would just not allow them to access your content.
I believe that when most people post public content on the Internet they intend it to be public including accessed by other search engines crawlers, etc.
Now we're in a horrible situation where just a few companies essentially own the Internet.
This is why Google can't index Facebook content or Twitter content even though it's public - they can't access it.