In a place with ubiquitous fast broadband, a compromise with regard to decompression speed might be ok.

If you're sending something to mars... 2 kb/sec might make sense.

Keep in mind that 2KB/s is on something like 3090... Massively power hungry even on the decompression.

You might be interested: Zstandard has an adaptive mode that attempts to do this.

Each file is really its own special theoretical multivariate space to find the best theoretical compression ratio. Usually finding the optimal or near-optimal compression is the hardest computationally.

And it's basically a perfectly cacheable computation if you have some sort of database mapping algorith + parameters to apply to a given file.

Decompression usually isn't as big a deal for performance.

Anyway an online database for a service that has actively tried to take a known common file (match on filesize, hash, etc) down to its best compressed representation so 1) we don't have to do it every time.

Cloud providers would presumably love these, since S3 probably has millions copies of many files and would 1) get to compress the file and 2) get to share its representation across all instances.

Apologies if this is a noob question, but could you improve it with training?

Considering all other things like GPU, code quality etc. equal, the speed here mainly depends on the size of the network. Larger networks result in better performance (logarithmic), but also slower speeds (linear). There are some tricks like pruning to create significantly smaller networks with similar-ish performance, but good models are often large.

Unrelated to your question, what nobody mentions is that the decompression side also needs to complete network. With 57M and 187M parameters, those files are going to be quite large, > 100MB for sure. That completely annihilates the performance wins for 1 time transfers.

As I understand it, when starting up (beginning of stream) - the network does not compress, then guesses the next byte at every turn determininstically from input so far.

Decoding can "read" the fist byte, then proceed to learn and guess the second byte and so on.

https://news.ycombinator.com/item?id=27244810

it's a very clever use of symmetry.

LZ78 used it "just" to build a dictionary, but the general idea of using symmetry is there.

(A fascinating variant on this is Semantic Dictionary Encoding, by Dr Michael Franz in his doctoral dissertation (1994), that used this symmetry to compress partial ASTs to cache partially generated code for runtime code generation)

LZ4HC is LZ4 compressed with the slow compressor to achieve better compression. It is decompressed using regular LZ4 decompressor.

They also compare with zstd and lzma, which are the open source competition.

I don't know what "specs" you think are missing. They do say which architectures are supported, what the memory use is, whether multi-threading is supported, etc.

No public price, true.

You can read the blogs of the people who work on this code to get a feel for the caliber:

https://cbloomrants.blogspot.com/

https://fgiesen.wordpress.com/

These are known people with known reputation. The product itself is well known in the industry.

I know shipping speeds are important, but it seems game developers could try harder with delta compressing their updates, because it's hard to believe a tens-of-GBs download is the best they could do when you already have a BluRay disk or something (unless it's new content, of course)

I wonder how much their bandwidth costs are and how quickly a reduction would pay itself.

Given the layers of encryption and obfuscation, and leaving aside the merits or otherwise of DRM in the first place, I can imagine it could be hard to do delta patching. At least, without making it easier to crack the DRM - e.g. you’d presumably have to decrypt the asset that needed patching, apply the delta, and re-encrypt/re-sign it all client side.

I'm not sure how viable this is today, or ever was: Many residential connections are heavily asymmetric. Players presumably aren't too excited either by the idea of continuing to seed while they already play online, with shitty ISP-provided routers suffering hundreds of milliseconds of bufferbloat whenever the uplink is saturated.

Yes, WoW used to have bit-torrent based update.

>Many residential connections are heavily asymmetric.

this is a US centric-view, though. For instance I have 300Mb up/down. At least for most of the Europe (and I suppose Asia), it's pretty common not to be restricted in any way.

I could leave a bit-torrent client non-stop and won't feel it. Like I've said: alas. There is some stigma associated with p2p/torrents.

In the Philippines, 15 Mbps down / 3 Mbps up ADSL with a static public IP is not something that a foreigner could easily buy. Globe Telecom in Cebu asked for evidence that I have sufficient income to pay for this (2600 PHP per month), and their rules only allow documents from the Philippines, so my bank statement was rejected. I had to talk to their higher-level manager to negotiate an exception. And their default router has 2 seconds of bufferbloat, so I bought a TP-Link Archer C7 and installed OpenWRT to compensate.

I live in Europe and I have a 1000/0.5 Mbit/s connection.

No more ACKs? No more DL!

I've seen 20:1 offers on cable providers, but it's mostly marketing BS, as you're unlikely to get full download speed in most circumstances due to upload limitations.

I do reach the full downlink speed though due to (presumably) ACK compression in the modem and ACK synthesis/reexpansion in the CMTS. You are right that otherwise such a high ratio would not be feasible with TCP.

IME, the line starts to choke when you actually want to send requests and they compete with ACKs and your internet stutters and crawls in both direction. Though that was with a slower connection - it's harder to fill a bigger pipe.

BTW, is the ACK compression counted for consumers benefit?

While asymmetric connections make sense for many residential use cases, P2P is not one of them.