News for hackers. Stuff that matters.
Re:Libgen Storage Decentralization on IPFS by segfaultbuserr on Wednesday November 25, 2020 @09:25AM (#25209501)
The IPFS mirror really gives a substantial performance boost. In the past, the files had to be fetched from a tired server far away in Russia, the speed can be as low as 0.5 Mbps in my experience (depends on your ISP and connectivity), now it can be lightning fast thanks to P2P, 2 Mbps+, more than adequate to download a 100 MiB+ file. You don't even have to install IPFS (but you probably should, to maintain decentralization), books can be downloaded straight via HTTPS from Cloudflare's IPFS reverse proxy (called a "gateway"), cloudflare-ipfs.com.

The IPFS network itself cannot be taken down, but I wonder when will the publishers discover IPFS web gateways, and start filing DMCA takedown requests against them, and what are the gateways gonna do? Will Cloudflare implement a huge copyright blocklist or even terminate its service (or worse, send a list of IP addresses to Prentice Hall or Elsevier)? What about the semi-official ipfs.io gateway? And the smaller independent IPFS gateways (there are currently 10+ on the public web)? Will they be DMCAed out of existence?

Re:Libgen Storage Decentralization on IPFS by traverseda on Wednesday November 25, 2020 @10:03AM (#25209933)
In my experience IPFS is only fast when you're using it through something like Cloudflare's IPFS proxy (which is basically a caching proxy). I haven't found IPFS to be actually usable any time I've tried running a node myself.

Is that unusual?

Re:Libgen Storage Decentralization on IPFS by magila on Wednesday November 25, 2020 @11:24AM (#25210888)
This has been the state of IPFS for years now. The protocol and client haven't been developed to where they can work reliably in a real P2P scenario. IPFS only appears to work because most requests go strait to one of the big gateways which already has the content cached.

Development of IPFS slowed to a crawl when most of Protocol Labs switched to working on Filecoin. It remains to be seen if either Filecoin or IPFS will ever be viable for their intended purpose.

Re:Libgen Storage Decentralization on IPFS by derefr on Wednesday November 25, 2020 @11:54AM (#25211293)
IPFS is the engine of Filecoin, though. Filecoin just incentivizes IPFS nodes to store things. So further IPFS development should still be necessary from the perspective of making Filecoin development/adoption happen.

(I say this, and yet I know it isn't true in practice. But why isn't it true?)

Re:Libgen Storage Decentralization on IPFS by nwah1 on Wednesday November 25, 2020 @12:34PM (#25211802)
Pump up Filecoin price. Sell Filecoin. Move to Fiji. Don't do any coding while in Fiji
Re:Libgen Storage Decentralization on IPFS by perlwle on Wednesday November 25, 2020 @08:04PM (#25215978)
Ha. I was super surprised that my dad(almost 70) mentioned IPFS to me the other day. It turns out the people he know are running or thinking of running IPFS nodes because of filecoin incentive. Apparently, the people don't want to miss another bitcoin opportunity.

filecoin is also promoting in China. https://mp.weixin.qq.com/s/T0Qt2CEP7QsA6Zy9Vzvdkw

Re:Libgen Storage Decentralization on IPFS by dsun179 on Wednesday November 25, 2020 @12:24PM (#25211675)
Yes, this is still the case. For a file which is on exactly one other computer, you have about 45 seconds lookup time before the download can start.
Re:Libgen Storage Decentralization on IPFS by georgyo on Wednesday November 25, 2020 @11:54AM (#25211276)
I'm not sure you are right about this. The following link will stream big buck bunny directly from the network with full P2P support. It is not using a gateway node for the video content.

https://bbb.fu.io/

You can run the following in the console to see all your peers

    for await (const peer of await node.swarm.peers()) { console.log(peer.addr.toString()) }
The console currently get's spammed pretty hard because they haven't implemented filtering out plain ws (non-wss) peers if you are connecting https. But it will use webrtc to find peers, and find all wss nodes in the network.
Re:Libgen Storage Decentralization on IPFS by magila on Wednesday November 25, 2020 @01:01PM (#25212102)
For me at least the vast majority of the data is being downloaded from various servers under bootstrap.libp2p.io or preload.ipfs.io. These are clearly dedicated servers not true peers.

The contact info for these servers appears to be retrieved via the DNS rather than the DHT. This isn't surprising because browser clients can't participate in the DHT, but the DHT is currently one of the biggest weak points of IPFS. IPFS's DHT is so chatty that it consumes over a megabit of throughput just to maintain an idle node. Lookups are excruciatingly slow and often fail completely.

Re:Libgen Storage Decentralization on IPFS by georgyo on Wednesday November 25, 2020 @01:07PM (#25212198)
Sorta, a peer can advertise dns addresses, and the browser can participate in the dht as a client. The example above uses webrtc-star for browser to browser discovery.

    $ ipfs dht findpeer 12D3KooWJxNHY6zE1KnzFdBJrTMCbhCVNtSLvjp7qR5Wsp49DnUC
    /dnsaddr/ipfs.scalable.io
    /dns4/ipfs.scalable.io/udp/4001/quic
    /dns6/ipfs.scalable.io/tcp/443/wss
    /dns6/ipfs.scalable.io/udp/4001/quic
    /dns4/ipfs.scalable.io/tcp/443/wss
Re:Libgen Storage Decentralization on IPFS by haywirez on Thursday November 26, 2020 @03:42AM (#25218112)
Took a good 10 seconds until the video started playing. Anything over a second is too long in terms of UX expectations.
Re:Libgen Storage Decentralization on IPFS by StavrosK on Thursday November 26, 2020 @07:48AM (#25219543)
I have to agree. I've had cases where I had to pin data that was on the node already and IPFS just hung the pin for minutes, failed, and then the GC removed the data later on. This is just one of the issues I had, other things include IPFS not being able to discover content on nodes that were already connected to it, IPFS just failing all my queued pins at once for some reason, etc.

I regret ever building a pinning platform on IPFS. The worst thing is that, when I expressed my frustration on Twitter about an IPFS release completely breaking IPNS updates, the founder called me a Karen.

Re:Libgen Storage Decentralization on IPFS by the_duke on Wednesday November 25, 2020 @10:23AM (#25210125)
No.

The last time I tried it (maybe 9-12 months ago) it was a real resource hog (including choking the network) and really slow.

Re:Libgen Storage Decentralization on IPFS by swills on Wednesday November 25, 2020 @04:32PM (#25214540)
Are you opening the required ports (4001, 8080)? I found it was like that too until I did opened the ports.
Re:Libgen Storage Decentralization on IPFS by segfaultbuserr on Wednesday November 25, 2020 @10:15AM (#25210052)
Interesting, didn't notice that before. I believed my downloads are too obscure to be cached by Cloudflare, but I may be wrong. I'll do a comparison once I fixed my server at home...
Re:Libgen Storage Decentralization on IPFS by tutfbhuf on Wednesday November 25, 2020 @11:04AM (#25210642)
IPFS gateways are similar to Tor exit relays, they just route the traffic to the public internet. Tor exit relays still exist, and haven't been DCMAed out of existence so far.

One could also imagine IPFS over Tor.

Re:Libgen Storage Decentralization on IPFS by segfaultbuserr on Wednesday November 25, 2020 @03:05PM (#25213667)
Tor exit nodes are forward proxies, they are used by the client-side to initiate connections to the public Internet but do not accept connections from the public Internet. You cannot host an Internet-facing webserver on a Tor exit (Onion Service exists solely within the Tor network), DMCA is not an issue, they do not act as a point of content distribution, there is nothing to take down. The main issue is the abusive outgoing traffic, which is seen as unavoidable and outweighed by Tor's greater benefits, so the existence of Tor exits is justified.

IPFS gateways (and Tor-to-Web) are reverse proxies, they are configured at the server-side to accept connections from the public Internet and route traffic to IPFS (and Tor), and they distribute content to the public Web. As far as I can see, this is a problem, someone can send DMCA notices, asking the gateway operators to take files down from the Web. The only protection I see is DMCA Section 512 (a.k.a Safe Harbor Provision), the operators are not liable for things solely hosted on IPFS, which is good, but gateways must comply with the takedown notices. So I think it's entirely possible for Cloudflare to put a huge blocklist on its IPFS gateway, and for smaller gateways that don't have the necessary resources to handle the requests, be DMCAed out of existence entirely. In the 2000s, RIAA launched campaigns against eD2k servers and BT trackers, including the use of honeypot servers to collect information. RIAA was not ultimately successful, but it created a major short-term disruption (eventually new servers would always appear in a different jurisdiction). A similar campaign against IPFS gateways sound possible if the publishers decided to launch an aggressive crackdown like the RIAA. The question is only a matter of whether the decision is made.

I'm not saying that IPFS requires the use of IPFS gateways, they don't, and in a sense, gateways are counterproductive for the decentralization goal. But they do currently provide an useful service for the Web, and I'm just speculating whether a major disruption is possible.

Re:Libgen Storage Decentralization on IPFS by tutfbhuf on Thursday November 26, 2020 @02:45AM (#25217827)
Thx segfaultbuserr for the detailed answer. I was wrong and I agree with you that IPFS gateways and Tor exits are two different things.

Well, then we might need something like IPFS-over-Tor, using Tors anonymity features to make it more safe for operators.

Re:Libgen Storage Decentralization on IPFS by mintplant on Wednesday November 25, 2020 @01:54PM (#25212842)
IPFS-over-Tor has been trapped in development hell for 5 years.

https://github.com/ipfs/notes/issues/37

Re:Libgen Storage Decentralization on IPFS by segfaultbuserr on Wednesday November 25, 2020 @02:32PM (#25213272)
If IPFS-over-Tor is in the development hell, IPFS-over-I2P is in the development limbo.

https://github.com/ipfs/notes/issues/124

Re:Libgen Storage Decentralization on IPFS by georgyo on Wednesday November 25, 2020 @11:15AM (#25210757)
You're almost right. The IPFS gateway nodes also cache the content. IE, they are both storing, hosting, and sharing to other IPFS nodes.
Re:Libgen Storage Decentralization on IPFS by mpfundstein on Wednesday November 25, 2020 @12:28PM (#25211719)
how can one validate that the cached version is indeed bit equal to the one stored within ipfs?
Re:Libgen Storage Decentralization on IPFS by georgyo on Wednesday November 25, 2020 @12:39PM (#25211853)
The address of the file is the hash of the file. (Really the hash of the DAG of the file)

So you could validate it if you wanted to. Because the URL _is_ the hash, if you did validate it you could be very sure that it is correct.

If a webserver hosts a file, how can you be sure that file is correct? Sometimes the ship a sums file next to it. At best this can find corruption, but it would not find malicious files.

Re:Libgen Storage Decentralization on IPFS by openfuture on Wednesday November 25, 2020 @12:37PM (#25211832)
You hash it, everything is content addressed.
Re:Libgen Storage Decentralization on IPFS by jakobdabo on Wednesday November 25, 2020 @12:00PM (#25211360)
Am I understanding it right that DMCA takedown requests apply to a URL?

Maybe IPFS can introduce a feature to generate one-time URLs, e.g. `oneTimeURL = f(realURL);`.

Websites then can generate new URLs for example every hour or even for each client request. Copyright bots will file a DMCA request to a URL which only they have.

Re:Libgen Storage Decentralization on IPFS by ALittleLight on Wednesday November 25, 2020 @06:22PM (#25215420)
I believe this is similar logic to what MegaUpload used. i.e. They would allow users to easily create links to content, and if DMCA'd would invalidate the links but leave the content alone (and the users free to generate arbitrarily many more links to keep sharing copyrighted content).

It kind of worked for MegaUpload. Their founder got rich and a version of the company still exists. However, the founder is also gradually losing a long court battle and may find himself extradited to the US where he will face significant jail time.

Re:Libgen Storage Decentralization on IPFS by judge2020 on Wednesday November 25, 2020 @12:30PM (#25211764)
In general DMCA takedown requests reference a URL to locate infringing content, but the actual requests ask that the content itself be removed and not just the URL.
Re:Libgen Storage Decentralization on IPFS by matheusmoreira on Thursday November 26, 2020 @09:30PM (#25225111)
They'll probably lobby the governments for more laws.

It's a politico-technological arms race. Governments make laws. People don't agree with them so they create technology that subverts the government. The state must then become even more tyrannical just to maintain the same level of power. We'll end up with either an uncontrollable population or a totalitarian government.

Re:Libgen Storage Decentralization on IPFS by an_opabinia on Wednesday November 25, 2020 @10:43AM (#25210378)
> send a list of IP addresses to Prentice Hall or Elsevier

I don’t know, do you think nobody has noticed LibGen by now? Do you think the reason they’re still running is because of some jurisdiction issue? Even the lawyers would be wrong on that one.

Re:Libgen Storage Decentralization on IPFS by segfaultbuserr on Wednesday November 25, 2020 @03:35PM (#25214012)
> do you think nobody has noticed LibGen by now? Do you think the reason they’re still running is because of some jurisdiction issue?

I'm not sure about the implication of your words. But, clearly, Elsevier has already tried to take legal actions against LibGen previously, and once in a while, LibGen's domain names are still being blocked and routinely replaced, just like Sci-Hub. I'd still say the reason it's still running is a jurisdiction issue, or let's say it's a geopolitical issue, it's using Russia to shield itself away from US influences.

But an IPFS gateway hosted in the United States by Cloudflare does not enjoy these legal and political benefits.

Re:Libgen Storage Decentralization on IPFS by mathnmusic on Wednesday November 25, 2020 @10:13AM (#25210022)
And what about when browsers (like Brave) start supporting IPFS natively, and gateways are not needed any more?
Re:Libgen Storage Decentralization on IPFS by a1369209993 on Wednesday November 25, 2020 @07:03PM (#25215652)
Browsers still don't support (better-established, though I'm not sure older offhand) TOR or bittorent yet, so I'm not terribly optimistic.