Re:Xz format inadequate for long-term archiving
by saurik on Saturday October 22, 2016 @01:58PM
(#12769746)
> The article claims xz was chosen by prominent open source projects due to hype. No, it was chosen because of its favorable size/speed tradeoffs.
No: that's why the LZMA compression format was chosen, and why the .lzma container format temporarily became popular as people started using lzma-tools. The move to xz, which results in slightly larger files for seemingly arbitrary reasons and which technically provides one benefit (seekability) almost everyone defeats (by compressing a tar file) while using the same relying on the same underlying compression algorithm is less-well motivated.
Re:Xz format inadequate for long-term archiving
by jl6 on Saturday October 22, 2016 @03:04PM
(#12770079)
Were there many prominent open source projects using the LZMA format before the creation of xz? Debian, for example (which is where this particular flamefest originates), adopted[0] xz the year following the stable v5.0.0 of XZ-Utils[1].
My impression was that the whole 7zip->p7zip->lzma_alone->xz journey was concluded before any "xz hype" began.
[0] https://lists.debian.org/debian-devel-announce/2011/08/msg00...
Re:Xz format inadequate for long-term archiving
by saurik on Monday October 24, 2016 @03:12PM
(#12781864)
I don't quite understand the timeline you are drawing here but the Debian Package Manager is a perfect example of a prominent open source project which used the LZMA format before the creation of xz. They recently shifted to "deprecate" the LZMA format, a change which I have removed from the negligible fork of dpkg I ship to the tens of millions of people running Cydia.
Re:Xz format inadequate for long-term archiving
by LukeShu on Saturday October 22, 2016 @01:03PM
(#12769462)
Regarding #4: Padding out a file is popular because of tape drives. You'll typically write the archive to the an appropriately sized tape raw, without a filesystem (which is just another format you have to maintain a parser for). So because the file will be smaller than the tape, there will be garbage at the end of the tape. Because we wrote the file to the tape directly, there is no way of knowing where it ends, and a tool reading the file will need to just deal with the garbage data.
Re:Xz format inadequate for long-term archiving
by jl6 on Saturday October 22, 2016 @01:28PM
(#12769585)
Doesn't one typically write to tape drives with tar, rather than writing a raw file?
Re:Xz format inadequate for long-term archiving
by LukeShu on Saturday October 22, 2016 @01:44PM
(#12769667)
Indeed. This is why the tar format explicitly allows garbage data at the end. So then people started pondering all of the nifty or clever things they could do with tar files. And they didn't want to give it up when they started compressing the tar files.
Re:Xz format inadequate for long-term archiving
by zeta0134 on Sunday October 23, 2016 @12:57AM
(#12772037)
Couldn't you just reverse the order then, and create an xz.tar? Maybe I don't understand the benefit of taring the data first.
Re:Xz format inadequate for long-term archiving
by wojcech on Sunday October 23, 2016 @06:52AM
(#12772954)
Tar is just bundling into a single file AFAIK. There is a slight benefit depending on your compression tool to tar and then compress, because (AFAIK again)some tools compress files individually and then write them into a hierarchical file(I guess this is what xz does as well, since it's searchable?). If you tar first, these tools will work better, since they encode patterns found in all files instead if doing it per file(which means e.g. if there is a header once per file, that will get compressed in the tar.comorpress, not in the . compress)
Re:Xz format inadequate for long-term archiving
by baruch on Monday October 24, 2016 @07:19AM
(#12777921)
I've created sortedtar Brewster if this assumption and while it is correct tree benefit is mostly negligible except for some edge cases.
Re:Xz format inadequate for long-term archiving
by Spooky23 on Saturday October 22, 2016 @10:21PM
(#12771544)
Old man moment!
Kids these days don't appreciate having random addressable storage for archive/backup data!
Re:Xz format inadequate for long-term archiving
by the_mitsuhiko on Saturday October 22, 2016 @01:06PM
(#12769477)
Garbage data is not a problem since the length is known.
Re:Xz format inadequate for long-term archiving
by LukeShu on Saturday October 22, 2016 @01:16PM
(#12769526)
I do not understand the xz format enough to evaluate that claim myself, but TFA explicitly claims that garbage data is a problem.
Re:Xz format inadequate for long-term archiving
by marcosdumay on Saturday October 22, 2016 @03:20PM
(#12770156)
Hum, no. When you tar a set of files directly into a tape, you don't know the resulting tar size beforehand. Even less if you compress the result.
Re:Xz format inadequate for long-term archiving
by mnarayan01 on Saturday October 22, 2016 @03:27PM
(#12770201)
I would think you could just write the size at the end of the tape?
Re:Xz format inadequate for long-term archiving
by ikawe on Sunday October 23, 2016 @10:23AM
(#12773682)
When you're reading the data later, how would you know where to find "the end" if you don't already know the length?
Re:Xz format inadequate for long-term archiving
by mnarayan01 on Sunday October 23, 2016 @04:52PM
(#12775372)
You don't know the end of the data, but presumably(?) you know the end of the tape.
Re:Xz format inadequate for long-term archiving
by marcosdumay on Monday October 24, 2016 @05:07PM
(#12782908)
We are talking past each other here...
Making a backup with tar is done by typing something like that on bash:
> tar -c - dir1 dir2 dir3 > /dev/tape
That will (hopefully, I doubt I got the tar switches right) backup those dirs into the tape (that will actually have a weird name, not '/dev/tape').
Now, in practice Linux doesn't always know the size of a tape you inserted. But this is not the issue, if you accept the seeks needed for that, you'd better write at the beginning anyway.
Re:Xz format inadequate for long-term archiving
by kahrkunne on Saturday October 22, 2016 @01:17PM
(#12769531)
Considering neither lzip nor xz is a commercial product, I don't think your final point stands. To me, it only makes sense that the person who wrote a "competing" file format would have a strong negative opinion of xz
1. The article itself recommends that "a tool is supposed to do one thing and do it well". Well, xz is for compression, nothing else. If you want to check integrity, use a hashing tool such as sha256sum. If you want to recover from errors, generate parity files using par2 or zfec.
2. The article claims xz was chosen by prominent open source projects due to hype. No, it was chosen because of its favorable size/speed tradeoffs.
3. It's not clear what the author means by "long-term archiving". Archives professionals (as in, people who are actually employed by memory institutions) will tell you of many factors (all unmentioned here) that have a bearing on whether a file format (particularly a compressed one) is suitable for use in data preservation.
4. The section on trailing data is particularly bizarre. By claiming that xz is "telling you what you can't do with your files", it seems that the author considers it perfectly reasonable to append arbitrary data to the end of a file and expect it to continue functioning. Is my normality detector off today or is this just a wacky thing to want to do?
5. The article is written by the author of lzip (which it compares favorably to xz), but does not disclose this. Admittedly it is hosted on the lzip site, but overall the article comes across as an opinionated hit-piece designed to sow doubt in a competitor.