News for hackers. Stuff that matters.
Re:Let us serve you, but don't bring us down by x-complexity on Monday May 29, 2023 @01:11AM (#36110899)
Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents.

The torrent protocol was meant to relieve this level of server load in mind.

Re:Let us serve you, but don't bring us down by dan_wood on Monday May 29, 2023 @01:41AM (#36111032)
ToRrEnTs ArE IlLeGaL ...
Re:Let us serve you, but don't bring us down by legends2k on Monday May 29, 2023 @02:00AM (#36111150)
Torrents aren't illegal inherently; the content downloaded are often copyright violations, yes, but using torrenting protocol itself isn't illegal. Most Linux distributions provide a torrent option to download their OS image file.
Re:Let us serve you, but don't bring us down by chaxor on Monday May 29, 2023 @02:40AM (#36111357)
Torrents are the backbone of a lot of academic research downloads. They're the absolute best way to distribute neural network weights, or large models used in physical science studies. That or IPFS. If you use huggingface or dockerhub for anything like this, seriously consider switching, as it will be extremely useful when those centralized systems inevitably fail.
Re:Let us serve you, but don't bring us down by userbinator on Monday May 29, 2023 @01:15AM (#36110914)
and loudly point towards downloading the files via torrents.

By "loudly", perhaps actually redirect to the .torrent instead, for those who trip the rate limit?

Re:Let us serve you, but don't bring us down by x-complexity on Monday May 29, 2023 @01:36AM (#36111003)
...that can work.
Re:Let us serve you, but don't bring us down by remram on Monday May 29, 2023 @05:33PM (#36118356)
A Link header would be more appropriate (maybe with rel="alternate"?)
Re:Let us serve you, but don't bring us down by maksimur on Monday May 29, 2023 @02:43AM (#36111377)
Torrents have the habit of disappearing when no users keep them alive. It happened to me enough times to be wary of such solution. If there's a way to keep them alive regardless of interest I'm all for it.
Re:Let us serve you, but don't bring us down by 5e92cb50239222b on Monday May 29, 2023 @02:53AM (#36111436)
How is that worse than serving files via HTTP? With HTTP archive.org is the only peer that can serve files to you, with BitTorrent it will be one of the (hopefully) many peers, and will degrade to the same level of redundancy as HTTP if all other peers leave.

BitTorrent also supports web seeds and they don't even really have to keep a full client running, just embed an HTTP link into the .torrent file.

Re:Let us serve you, but don't bring us down by chronogram on Monday May 29, 2023 @03:13AM (#36111527)
Have you tried an Archive.org torrent yet? It's backed by the servers, but has the advantages of selecting which parts of the archive you want and being verifiable and being able to have more bandwidth on popular archives. The Archive.org servers show up on the "HTTP sources" tab next to the "Peers" tab for me.
Re:Let us serve you, but don't bring us down by happywolf on Monday May 29, 2023 @03:16AM (#36111539)
Seeding peers or machines are meant to mitigate this issue.
Re:Let us serve you, but don't bring us down by nextaccountic on Monday May 29, 2023 @04:11AM (#36111832)
The very same machines that serve http can also seed torrents. If those machines go offline, they also take down the http endpoint
Re:Let us serve you, but don't bring us down by shepherdjerred on Monday May 29, 2023 @04:21AM (#36111893)
That's no worse than just relying on Arhcive.org to be sole provider
Re:Let us serve you, but don't bring us down by maksimur on Monday May 29, 2023 @04:23AM (#36111907)
Thanks everyone for your reply. It made me reconsider my opinion. Though the Reddit-style downvotes are unnecessary and somewhat nerve-wracking. Use them when they're actually necessary. Not as gate keeping or as a "don't agree" button or whatever frivolous thing is passing through your mind.
Re:Let us serve you, but don't bring us down by slim on Monday May 29, 2023 @06:19AM (#36112492)
consider seeding the torrents you download
Re:Let us serve you, but don't bring us down by jeroenhd on Monday May 29, 2023 @07:25AM (#36112900)
The torrents the archive provides are backed by web seeds served by the internet archive web servers.

This approach allows the internet archive to effectively rate limit hosts without making content unavailable entirely, and allows others to help carry the load.

Re:Let us serve you, but don't bring us down by ElongatedMusket on Monday May 29, 2023 @07:31AM (#36112939)
See https://webtorrent.io/faq for some ideas of doing what you're wishing
Re:Let us serve you, but don't bring us down by jlokier on Monday May 29, 2023 @09:56AM (#36114012)
This is what "proof of storage" or "proof of data availability" blockchain networks are for. They use economic incentives to continuously pay nodes a small amount to store some data and keep it available, and the cryptographic sampling mechanism ensures that less popular data must remain in the available dataset for nodes to be paid, even if it is rarely requested in full.
Re:Let us serve you, but don't bring us down by remram on Monday May 29, 2023 @05:35PM (#36118371)
If you are willing to spend money, there is no problem keeping content up with existing technology. Like always, blockchain solutions miss the problem entirely.
Re:Let us serve you, but don't bring us down by jlokier on Tuesday May 30, 2023 @01:38PM (#36127868)
It would be nice not to shoot the messenger. The problem described by maksumur is a real problem people encounter often - torrents for less popular large files that people only occasionally want, but do want or need, have a habit of disappearing, or being non-functional when you do find a seed. Popularity plays too strong a role.

As far as I know the proof-of-data-availability folks (whether using a blockchain or not) are the only folks trying to solve this problem with a serious technical solution at a cost scale lower than "rent your own server and have deep pockets for bandwidth" at the moment.

It's not about "willing to spend money", as if there is only one threshold to pass.

Torrents are used largely because they diffuse the cost of hosting, so that people (and organisations) don't need deep pockets to distribute large files to many others. There is an inevitable infrastructure cost but it's spread out more fairly among users.

The proof-of-data-availability stuff is similar, but takes it further so that a wider range of data stays publically available to whoever wants to download it, whenever they want, than it otherwise would be. It is really just a more fancy version of torrenting that is less prone to the excessive per-file popularity fluctuations that torrents suffer from. The objective is to lower costs compared with the current state of the internet, not add more. And it does not specifically require a blockchain.

If you know of something else tackling the problem I'd be love to hear about it. Torrent seed sites don't qualify, as they don't solve the problem: Most files aren't available on them either, and you have to pay for the more obscure content they do have.

Re:Let us serve you, but don't bring us down by remram on Tuesday May 30, 2023 @04:59PM (#36130711)
No, that still misses the mark entirely. We are in a thread about archive.org's server being hit too hard, someone proposed they should offer distributed downloads via torrent, and someone commented asking for "a way to keep them alive regardless of interest".

You are proposing a system based on financial rewards for hosting. Who pays those rewards, for those files in which there is no interest? If archive.org is to pay for it, we are back to square one. If people with an interest in download the files pay for it, no go, files with no interest go away. If you propose that people currently abusing the free service of archive.org, to the point of bringing it down, would pay a fee per download, you must be joking.

Re:Let us serve you, but don't bring us down by jlokier on Wednesday May 31, 2023 @09:52AM (#36138429)
> You are proposing a system based on financial rewards for hosting.

No, I'm not. You are incorrectly assuming that blockchains are necessarily financial or that cryptoeconomic incentive structure involves net pay to someone.

> If you propose that people currently abusing the free service of archive.org, to the point of bringing it down, would pay a fee per download, you must be joking.

I'm not proposing that.

People "pay" for hosting by participating in some amount of upload to offset their download, in order to be granted higher download rates. That is the same principle as BitTorrent has used since its inception: Upload is measured and download is traded for upload to ensure users choose to upload for a while.

The difference is that information is split and diffused in a different way, which ensure that some amount of upload bandwidth and temporary storage is available for less popular data as long as there are people participating in the network, mostly when they are downloading something more popular and providing some upload in exchange. The network power law helps by ensuring the long tail of less popular content needs relatively little "extra" bandwidth so it is not onerous on the users who, most of the time, are downloading and storing popular data.

No money needs to be involved.

Probably some financial things will emerge much like paid Torrent sites do at the moment for enhanced access which some people prefer. You can't prevent them, but they are not required for the network's operation nor required to access it.

Re:Let us serve you, but don't bring us down by npteljes on Tuesday May 30, 2023 @09:10AM (#36124449)
Torrent needs seeders the same as URLs need their original servers. There's a way to upgrade http to torrent though, like this: https://en.wikipedia.org/wiki/BitTorrent#Web_seeding
Re:Let us serve you, but don't bring us down by ranger_danger on Monday May 29, 2023 @03:52AM (#36111739)
Depending on the type of content you download, the chances of anyone else already seeding that might actually be zero.
Re:Let us serve you, but don't bring us down by andruby on Monday May 29, 2023 @10:09AM (#36114144)
Archive.org would also act as a seed, so at worst it’s similar to regular download traffic. At best there’s a few more people downloading and seeding.