fwessels

disclaimer: I am one of the co-authors pf the code as well as founder

Thank for for the feedback, that is nice to hear. And as for the business question, we plan on launching a Sneller Cloud offering. (Sneller founder here)

Sneller founder here: we do not have any non-AVX code so we cannot compare directly against that. But generally speaking our code always works on 16 lanes in parallel per core, so that gives a huge speed-up.

Albeit not using gas, you may want to check out https://github.com/SnellerInc/sneller -- it has about ~250 primitives written in AVX-512. (Sneller founder)

We have added both a cost and power consumption comparison to the blog post (including AMD EPYC as well), here is a snippet:

| Instance type | m6g.16xlarge | c5.18xlarge | c5.24xlarge | c5a.24xlarge |

| Architecture | Graviton2 | Skylake | Cascade lake | AMD EPYC |

| CPU(s) | 64 | 72 | 96 | 96 |

| RAM (GiB) | 256 | 144 | 192 | 192 |

| $/hour | $2.464 | $3.06 | $4.08 | $3.696 |

| Power usage (W) | ~115 | ~240 | ~240 | ~280 |

As per last week's announcement from AWS about the availability of AMD EPYC cpus, we have repeated the “single socket” test for the EPYC cpu as well.

The updated chart is included in the post or you can find it here: https://blog.min.io/content/images/2020/06/single-socket-per...

It clearly performs and scales a lot better than the Intel CPUs (no doubt also benefiting from the increased memory bandwidth) and at high core counts is very close to the Graviton2.

Furthermore, I agree that the “investment costs” for Amazon are almost minute, they might have already earned it back.

Check out the graph at https://github.com/fwessels/HashCompare that compares different hashing techniques.

And also we've done some predictions for collisions, see: https://github.com/fwessels/HashCompare/issues/1#issuecommen...

Yes, you can, but just naming the return variable takes care of this.

No, it is not an IR code such as in Java or .Net. At compile time it is translated into opcodes for the corresponding architecture that you are compiling for.

That is a good question but have never tried it. However, there is nothing "special" about the assembly as ported by c2goasm, so it should equally well as any other (hand-crafted) assembly.

S3 data transfers costs are an issue -- that's why you can host minio yourself at any hosting company, and save significantly (multiple times) on data transfer and storage costs.

Nice blog, and the tool would have helped indeed.

And also, memory allocation is not the most obvious topic in Go/Golang assembly...

The Golang assembly does not support all instructions, we've created a tool (see https://github.com/minio/asm2plan9s) to assist in translating instructions into their opcode equivalents.

No we did not compare to SHA512 (since we don't need it).

I would expect SHA512 to be 2x faster compared to the SHA256 software version, but that is still way slower than the ARM SHA extensions accelerated version.

No, there are no algorithm changes (otherwise the results would be different), it is just taking advantage of the ARM SHA accelerations when they are available.

No, compared to an ASIC it will still be significantly slower.

Go didn't help specifically, it is just that this package makes support for the ARM SHA instructions available for Golang.

The library is not yet open sourced but will be soon. Some internals about how s3git uses the BLAKE2 hashing algorithm are described in https://github.com/s3git/s3git/blob/master/BLAKE2.md which may be of interest.

That is correct (see also earlier reply), it is aimed at really large repos with primarily non-text blobs backed up cloud storage like S3, but doing so in the spirit of git and hence the name.

Thanks for the links. However s3git more applies the git philosophy/way of working to Cloud Storage rather than being compatible at a binary level.