--adapt[=min=#,max=#]
zstd will dynamically adapt compression level to perceived I/O conditions. Compression level adaptation can be observed live by using command -v. Adaptation can be constrained between supplied
min and max levels. The feature works when combined with multi-threading and --long mode. It does not work with --single-thread. It sets window size to 8 MB by default (can be changed manu‐
ally, see wlog). Due to the chaotic nature of dynamic adaptation, compressed result is not reproducible.Similarly, on a local speed test (SSD -> SSD), using a fixed compression level was much faster than --adapt.
"" note : at the time of this writing, --adapt can remain stuck at low speed when combined with multiple worker threads (>=2). ""
Leave wlog alone unless you're willing to store the value out of band and pass it in again during decompression.
hashLog, bigger number uses more memory to compress but is often faster.
chainLog smaller number compresses faster, but worse ratio.
In your use case monitoring general system utilization to identify bottlenecks might also help. My gut instinct is that you might already have hit a memory bandwidth limit for the platform, at which point REDUCING the hashLog until it fits within your intended performance budget might yield better bandwidth results. Reducing the chainLog value might have the same effect.
[1] https://en.wikipedia.org/wiki/TCP_congestion_control#TCP_BBR
I believe this is because it puts similar strings together in the tree gzip uses. clumpify.sh from bbtools is one example.
https://jgi.doe.gov/data-and-tools/software-tools/bbtools/bb...
You may also filter out duplicates and depending on what you do correct base errors or get rid of low complexity reads.
The bgzf format was invented for bioinformatics, and BAM files are bgzipped by default.
It blew people's minds when we couple implement huge projects that they could never get of the ground for years in just a matter of months because of 'this one cool trick'.
Man I miss Santa Cruz ಥ﹏ಥ worst mistake of my life to leave there.
Why not use some binary format like BSON? Compressing with gzip works but then you can't query it compressed
And text is easier to work with in any hacked up script.
There are some binary formats (BAM, I think), but people often prefer the text format anyway. When compressed, the size is pretty much the same
They do, but you need to uncompress to read the N-th letter
One trick I discovered is that tools like pigz can be used to both accelerate the compression step and also copy to cloud storage in parallel! E.g.:
There is a similar pipeline available for s3cmd as well with the same benefit of overlapping the compression and the copy.However, if your tools support zstd, then it's more efficient to use that instead. Try the "zstd -T0" option or the "pzstd" tool for even higher throughputs but with same minor caveats.
PS: In case anyone here is working on the above tools, I have a small request! What would be awesome is to automatically tune the compression ratio to match the available output bandwidth. With the '-c' output option, this is easy: just keep increasing the compression level by one notch whenever the output buffer is full, and reduce it by one level whenever the output buffer is empty. This will automatically tune the system to get the maximum total throughput given the available CPU performance and network bandwidth.