Nyan

Nice work - thanks for your efforts!

Yeah, the compute complexity for Reed Solomon is generally O(size_of_input * number_of_recovery_blocks)

If you don't specify the number of blocks, par2cmdline defaults to 2000, so at 25%, it's generating 500 parity blocks, which is obviously much slower than what you're generating with the other tools.

Having said that, PAR2 is generally aimed at use cases with hundreds/thousands of parity blocks, so it's going to be at a disadvantage if you're generating less than 10 - which your benchmark shows.

Thanks for doing that.

> par2cmdline[-turbo] encode: par2 c -r25 test

That command is rather unfair to PAR2 - you should add `-b48 -n2 -u` to make the comparison fairer.

PAR2 ain't exactly fast, particularly compared to GF8 focused formats, but the numbers you initially gave seemed wildly off, so I suspected the comparison wasn't really fair.

Ideally you should also be including the version of each tool used.

You should at least be benchmarking against par2cmdline-turbo instead (stock par2cmdline isn't exactly performance-oriented). Also, you need to list the parameters used as they significantly impact performance of PAR2.

Your benchmark also doesn't list the redundancy %, as well as how resilient it is against corruption.

One thing I note is that both ISA-L and zfec use GF8, whilst PAR2 uses GF16. The latter is around twice as slow to compute, but allows for significantly more blocks/shards.

Someone made this: https://github.com/manticore-projects/fpng-java

Replacing zlib might give you a few percentage points' worth of difference, whilst fpnge would likely be several times faster.

We've already got JPEG-XL as a good replacement format, but some feel that jeopardizing its adoption is a sensible cause.

Why not use fpnge?

If you search up guides on how to upload to Usenet, you'll often find most of the recommend creating split RARs.

Uploaders who just transfer scene releases without bothering to extract will of course stick to RAR, but there's a lot of content these days that isn't from scene groups.

Many older formats, ZIP included, didn't standardise on an encoding, so they just use whatever the local 8-bit encoding happens to be. And if your system charset is different from the ZIP creator's, you could get messed up filenames.

This is just another cost of sticking to legacy formats.

Hopefully that slowly stops being the standard: https://github.com/animetosho/Nyuu/wiki/Stop-RAR-Uploads

I follow the latest stuff but haven't seen anything I'd consider to be "a very large resurgence".

Personal opinion: whilst bad official translations certainly exist, most of the time, they're fine. There's usually stuff that could be improved (for example, songs often don't get included, typesetting etc), but as far as portraying the meaning, they generally work. And this is likely a key reason why fansubs died - the official subs are often in the "good enough" category.

Generally the translator will find a way to work around the lack of a direct equivalent, trying to convey as much of the meaning as possible.

Now you may think that these workarounds cause some meaning loss, and you'd be correct. However, the problem with this line of thinking is that this isn't unique to honorifics - there's a tonne of other stuff in the Japanese language which has no English equivalent (I don't know Italian, so can't say much on that front), and non-Japanese speaking anime fans are often unaware of these because fansubs never introduced those concepts (for example, purpose of -desu/-masu which can be used to show respect, somewhat like honorifics).

The reality is that a lot of nuance gets lost in translation, with or without honorifics; unfortunately, if nuance is important to you, there's no substitute to learning the language. As such, people who think honorifics are highly important generally give me the impression of blissful ignorance (i.e. don't know what they don't know).

Of course, you're welcome to have your own preferences regarding translations. But having some hybrid of Japanese/English subtitle to portray slightly more nuance (whilst still missing out on a lot), can seem weird (e.g. why is this particular aspect exposed, but not something else?) and isn't necessarily objectively "better" than a translation that just sticks to English.

From memory, the original Nyaa was nyaatorrents.org -> nyaa.eu -> nyaa.se

I'm not too familiar with JoJo, but that's more of an isolated case. These do exist of course, but they aren't representative of the majority.

If you look at the shows airing this season (or last season, if you prefer), the vast majority do not have any fansub releases. For those that do, they're typically minor edits of the official subs.

> look no further than the book and movie of Ready Player One. That's a fair comparison.

That's not even considered to be anime.

* https://myanimelist.net/search/all?q=ready%20player%20one&ca... * https://anidb.net/perl-bin/animedb.pl?show=animelist&adb.sea...

Even if it was, isolated incidents aren't a good representation of the larger picture.

Your experience may be slightly out of date.

Fansubbers these days are a tiny shadow of that they used to be. The vast majority of anime piracy these days consists of ripping official streams, which include subtitles.

Of course, if you're into older shows, you'll find many to be missing on the mainstream streaming platforms, and will often need to resort to piracy and fansubs to get that. But as anime becomes more mainstream, those interested in older stuff become a smaller slice of the market.

Also, due to more mainstream adoption, those interested in "cleaner rips, better encoding, etc" are dwindling. Whilst these people are still out there, you'll find the vast majority of people are perfectly happy with low quality streaming sites (also worth pointing out that these streams still offer vastly better video quality than what you typically got in the 90s).

Enforcement wise, I doubt they're targeting the hardcore fans - they're going after the mainstream, which is a big slice of the market.

A possibility might be that the bigger sites have paid more attention to opsec, and are harder to take down, whilst smaller players may be easier to intimidate.

Current Nyaa (nyaa.si) has been targeted [https://torrentfreak.com/mpa-lawyers-are-trying-to-shut-down...], but efforts haven't been successful so far.

> I guess because these are commercial?

Highly unlikely - it's not like rights holders get anything regardless of commercial status. Also, Nyaa runs ads on their NSFW site, so it's commercial in that respect.

> It's jumped several domains in the past

One thing to note is that there's actually two Nyaa sites, so it's important to not confuse the two.

The original jumped several domains, and closed down in May 2017 (last domain was nyaa.se). The "current Nyaa", nyaa.si, is a clone site that launched shortly after, and hasn't changed any domains.

With the original, I recall the domains jumped were linked to various legislative changes in Europe, with the shutdown in 2017 likely due to similar reasons.

Very useful. In fact, it speeds up a single instance (i.e. not taking advantage of SIMD) of MD5 by 20%: https://github.com/animetosho/md5-optimisation#x86-avx512-vl...

Faster PNG en/decoders exist though, it's probably just libpng being the most commonly used, but not extensively optimised.

I didn't infer 15% from the way it was written there. But most platforms these days have some form of CRC32 "acceleration". Adler32 is easy to compute so I'm even less concerned there.

I spent a bunch of time optimising the code in fpnge (https://github.com/veluca93/fpnge), which is often notably faster than fpng (https://github.com/nigeltao/qoir/blob/5671f584dcf84ddb71e28d...), yet checksum time is basically negligible.

Having said that, the double-checksum aspect of PNG does feel unnecessary.

I see. Yeah, PAR3 tries to address the 'lots of files' scenario better than PAR2.

But updates are problematic. You could 'delete' parts of a recovery file if the data is present. However a file being updated typically means that the original data (before the update) is no longer present, meaning you can't really 'delete' that part of the recovery to replace it with the new data. You could try and retrieve the old data by recovering it, but at this point, it may just be easier to recreate the entire PAR set again.

If it's just adding to the recovery set, PAR3 does have provision for that.

Ah, PAR makes the most sense for archival material like that. What were you looking for in the PAR format that'd make more sense for this use case?

Sorry if I'm dense, but what does "this use-case" exactly refer to here?

PAR is somewhat unwieldy to use. In addition to needing to explicitly create it (and it not being particularly fast, on a large enough data set), PAR2s can't be 'updated'. The PAR3 spec allows for some limited updating, but it's far from ideal.

It often makes more sense for the file system to deal with ECC in my opinion. PAR probably makes more sense for archived files that aren't expected to change, but may be moved across file systems.

PAR2 handles subfolders by the way, just not empty folders.

As someone who has been recently optimising fpnge, Adler32 computation is pretty much negligible regarding overall runtime. The Huffman coding and filter search take up most of the time. (IIRC fpng doesn't do any filter search, but Huffman encoding isn't vectorized, so I'd expect that to dominate fpng's runtime)

Nice!

> For deferring the madd specifically, using two separate sum2 vectors and splitting the `mad` vector into two

Actually, the idea was to accumulate into 16-bit sums, and only do madd to 32-bit every 4 loop cycles. I'm not sure splitting it up like that actually helps, since the latency can be easily hidden by an OoO processor, and could actually be detrimental adding more uOps.

One thing to note is that you've got a dependent add chain on sum2_v, so using two independent sums instead of one could help.

> Plus I am sure there are many other opportunities to optimize this I have not thought of :)

Other implementations I've seen don't go any further, e.g. https://github.com/zlib-ng/zlib-ng/blob/develop/arch/x86/adl... https://github.com/veluca93/fpnge/blob/9a9fc023870bacd06674f...

So perhaps as you allude to, it isn't really worth it.

Nicely done.

> There is still a lot of room to micro-optimize both the avx and avx64 implementation

I personally couldn't see much - perhaps aligning loads and defering `_mm256_madd_epi16` are the only ideas that come to mind. What did you have in mind?

Didn't know the practicality of this in the demoscene - thanks for the info!