News for hackers. Stuff that matters.
Re:Xz format inadequate for long-term archiving by Someone on Saturday October 22, 2016 @11:45AM (#12769125)
"the right way of implementing binary filters is to write a preprocessor that applies the filter to the data before feeding them to the compressor. (See for example mince)."

So, what does mince do? https://github.com/Kingsford-Group/mince/blob/master/README.... says:

"Mince is a technique for encoding collections of short reads."

That doesn't tell me much.

Also, http://bioinformatics.oxfordjournals.org/content/early/2015/...:

"We present a novel technique to boost the compression of sequencing that is based on the concept of bucketing similar reads so that they appear nearby in the file"

I'm still not sure I understand that fully, but guesstimate that it shuffles file content in such a way that it compresses so much better that it more than offsets the space needed to store the information on how to deshuffle the decompressed data. Is that correct?

Or does it require you to specify the parts and discards the reshuffling information, assuming you aren't interested in the order?

Re:Xz format inadequate for long-term archiving by yuubi on Saturday October 22, 2016 @04:00PM (#12770326)
I think the latter. According to the decompression instructions in README.md, "The reads will NOT be in the same order as in the original file".

It sounds like it's meant results of DNA sequencing processes that produce sequences for random small pieces of the molecule.

Re:Xz format inadequate for long-term archiving by john_reel on Saturday October 22, 2016 @05:56PM (#12770735)
Sure sounds a lot like the Burrows--Wheeler transform...