Re:Stopping the Internet of Noise – A useful internet back a...
by TeMPOraL on Wednesday July 05, 2017 @05:25AM
(#14700974)
Forget Github. It might be just a regular git repo on someone's server. IANAL, but could companies really go against a person hosting the source code of a software that scraps a website?
Even if, that's probably solvable (by turning it into cat-and-mouse game, TPB/SciHub-style). Have a non-turing-complete set of conversion rules that do scrapping -> standardized-data-format via a general-purpose scrapper. Distribute those through P2P. There might be problems with malicious actors, but generally this should be possible to implement in a way that's mostly transparent for end users.
> And you still need to account for upstream changes.
Upstream changes will never be too extreme to handle manually, if we're talking scrapping solutions - because any such extreme changes would also piss off the regular browser users.
Even if, that's probably solvable (by turning it into cat-and-mouse game, TPB/SciHub-style). Have a non-turing-complete set of conversion rules that do scrapping -> standardized-data-format via a general-purpose scrapper. Distribute those through P2P. There might be problems with malicious actors, but generally this should be possible to implement in a way that's mostly transparent for end users.
> And you still need to account for upstream changes.
Upstream changes will never be too extreme to handle manually, if we're talking scrapping solutions - because any such extreme changes would also piss off the regular browser users.