News for hackers. Stuff that matters.
Re:pigz: A parallel implementation of gzip for multi-core ma... by jakobnissen on Tuesday October 18, 2022 @04:11AM (#33244377)
Bioinformatician here. Consider using bgzf instead. It's fully gzip backwards compliant (it is a subset of gzip), but de/compression can be implemented to be much faster, and it's also much easier to paralellize. Compression rates are slightly lower, i.e it creates larger files.

The bgzf format was invented for bioinformatics, and BAM files are bgzipped by default.

Re:pigz: A parallel implementation of gzip for multi-core ma... by jiggawatts on Tuesday October 18, 2022 @06:17AM (#33245140)
These days, skill is not about how you use tools in IT, but simply knowing that a tool exists.
Re:pigz: A parallel implementation of gzip for multi-core ma... by ROTMetro on Tuesday October 18, 2022 @12:50PM (#33250345)
Hasn't that always been the case? When I lived in the Bay Area and networked, and was exposed to so much of how people did things, I was able to land much more contracts because I could come up with simple existing solutions. Once I moved away and wasn't exposed to cool people doing cool problem solving things, I had to switch into more bog standard consulting.

It blew people's minds when we couple implement huge projects that they could never get of the ground for years in just a matter of months because of 'this one cool trick'.

Man I miss Santa Cruz ಥ﹏ಥ worst mistake of my life to leave there.

Re:pigz: A parallel implementation of gzip for multi-core ma... by elmolino89 on Tuesday October 18, 2022 @08:12AM (#33246040)
bgzf has a clear advantage when it comes to sorted BED/VCF/GTF formats, especially if one does index these. But frankly I have no idea if it may improve IO times for the fastq files when read by say bwa or another mapper. Do you have any experience with that? I have seen some mapping times improvement using fastqs with clustered reads (by clumpify).
Re:pigz: A parallel implementation of gzip for multi-core ma... by jakobnissen on Tuesday October 18, 2022 @09:45AM (#33247025)
Yes, bgzf can be much faster in practice. Both because the underlying gzip implementation can be simplified (see e.g. libdeflate), and because it can be more effectively parallelised. It doesn't matter if it's IO-bound of course, but compression rarely is.