News for hackers. Stuff that matters.
Re:Hackers now use ZIP file concatenation to evade detection by wahern on Friday November 15, 2024 @07:18PM (#42152909)
From a security perspective, and as a programmer, I've never liked ZIP files precisely because there are two mechanisms to identify the contents, the per-file header and the central directory. When you're defining a format, protocol, or w'ever, ideally there should be a single source of truth, a single valid & useable parse, etc; basically, the structure of the data or process should be intrinsically constraining. There shouldn't be a pathway for multiple implementations to produce different functional results, and ZIP archives are in my mind the archetype for getting this wrong. tar files aren't ideal, but in the abstract (ignoring issues with long file names) they don't have this problem. (tar files don't support random access, either, but better to rely on something suboptimal than something that's fundamentally broken.)

A similar security problem, though not as fundamentally baked into the format, is MIME parsing. The header section is supposed to be delimited from the body by an empty line (likewise for nested entities). But what if it's not? For better or worse, Sendmail was tolerant of the absence of an empty line and treated as headers everything up to the first line that didn't parse as a header or header continuation.[1] Later systems, like Postfix, originally copied this behavior. But Microsoft Exchange and Outlook are even more tolerant, yet in a much more horrendous way, by parsing as a header anything that looks like a Content-Type or related header immediately after the first empty line. They have similar hacks for other, similar violations. So today, depending on the receiving software, you can send messages that appear differently, including having different attachments. It's a security nightmare!

I not a Postel's Law hater, but ZIP archives and Microsoft's MIME parsing behaviors are just egregiously wrong and indefensible. And even if you think the Robustness Principle is inherently bad policy, you still have to design your formats, protocols, and systems to be as intrinsically constraining as possible. You can't rely on vendors adhering to a MUST rule in an RFC, unless it's unquestioningly crystal clear what the repercussions will be--everybody else will (because it's the natural and convenient thing to do) reject your output as trash and drop it on the floor immediately so violations never have a chance to get a foothold.

[1] MTAs don't necessarily need to care about MIME parsing, but Sendmail eventually gained features where parsing message contents mattered, setting the de facto norm (for those paying attention) until Microsoft came along.

Re:Hackers now use ZIP file concatenation to evade detection by failbuffer on Friday November 15, 2024 @08:34PM (#42153375)
In a similar vein, HTTP header smuggling attacks exploit differences in header parsing. For instance, a reverse proxy and a web server might handle repetition of headers or the presence of whitespace differently.
Re:Hackers now use ZIP file concatenation to evade detection by kevin_thibedeau on Friday November 15, 2024 @10:34PM (#42153922)
The central directory allows zip archives to be split across multiple files on separate media without needing to read them all in for selective extraction. Not particularly useful today but invaluable in the sneakernet era with floppies.
Re:Hackers now use ZIP file concatenation to evade detection by thrdbndndn on Friday November 15, 2024 @11:43PM (#42154324)
Still useful today.

Try to transmit s 100G file through any service is usually a pain especially if one end has non-stable Internet.

Re:Hackers now use ZIP file concatenation to evade detection by SubiculumCode on Friday November 15, 2024 @11:51PM (#42154383)
I recently had to do this with about 700Gb, and yeah OneDrive hated that. I ended up concatenating tars together.
Re:Hackers now use ZIP file concatenation to evade detection by rakoo on Saturday November 16, 2024 @01:46AM (#42154890)
That's a very bad way of solving that issue. If transmission is a problem, either use a proper retry-friendly protocol (such as bittorrent) or split the file. Using hacks on the data format just leads to additional pain
Re:Hackers now use ZIP file concatenation to evade detection by meehai on Saturday November 16, 2024 @02:14AM (#42155000)
couldn't agree more!

We need to separate and design modules as unitary as possible:

- zip should ARCHIVE/COMPRESS, i.e. reduce the file size and create a single file from the file system point of view. The complexity should go in the compression algorithm.

- Sharding/sending multiple coherent pieces of the same file (zip or not) is a different module and should be handled by specialized and agnostic protocols that do this like the ones you mentioned.

People are always doing tools that handle 2 or more use cases instead of following the UNIX principle to create generic and good single respectability tools that can be combined together (thus allowing a 'whitelist' of combinations which is safe). Quite frankly it's annoying and very often leads to issues such as this that weren't even thought in the original design because of the exponential problem of combining tools together.

Re:Hackers now use ZIP file concatenation to evade detection by TeMPOraL on Saturday November 16, 2024 @02:57AM (#42155151)
Well, 1) is zip with compression into single file, 2) is zip without compression into multiple files. You can also combine the two. And in all cases, you need a container format.

The tasks are related enough that I don't really see the problem here.

Re:Hackers now use ZIP file concatenation to evade detection by meehai on Saturday November 16, 2024 @03:08AM (#42155187)
I meant that they should be separate tools that can be piped together. For example: you have 1 directory of many files (1Gb in total)

`zip out.zip dir/`

This results in a single out.zip file that is, let's say 500Mb (1:2 compression)

If you want to shard it, you have a separate tool, let's call it `shard` that works on any type of byte streams:

`shard -I out.zip -O out_shards/ --shard_size 100Mb`

This results in `out_shards/1.shard, ..., out_shards/5.shard`, each of 100Mb each.

And then you have the opposite: `unshard` (back into 1 zip file) and `unzip`.

No need for 'sharding' to exist as a feature in the zip utility.

And... if you want only the shard from the get go without the original 1 file archive, you can do something like:

`zip dir/ | shard -O out_shards/`

Now, these can be copied to the floppy disks (as discussed above) or sent via the network etc. The main thing here is that the sharding tool works on bytes only (doesn't know if it's an mp4 file, a zip file, a txt file etc.) and does no compression and the zip tool does no sharding but optimizes compression.

Re:Hackers now use ZIP file concatenation to evade detection by kd5bjo on Saturday November 16, 2024 @07:51AM (#42156150)
The key thing that you get by integrating the two tools is the ability to more easily extract a single file from a multipart archive— Instead of having to reconstruct the entire file, you can look in the part/diskette with the index to find out which other part/diskette you need to use to get at the file you want.
Re:Hackers now use ZIP file concatenation to evade detection by canucker2016 on Saturday November 16, 2024 @12:05PM (#42157450)
Don't forget that with this two-step method, you also require enough diskspace to hold the entire ZIP archive before it's sharded.

AFAIK you can create a ZIP archive saved to floppy disks even if your source hard disk has low/almost no free space.

Phil Katz (creator of the ZIP file format) had a different set of design constraints.

Re:Hackers now use ZIP file concatenation to evade detection by shagie on Saturday November 16, 2024 @09:46AM (#42156692)
In unix, that is split https://en.wikipedia.org/wiki/Split_(Unix) (and its companion cat).

The problem is that on DOS (and Windows), it didn't have the unix philosophy of a tool that did one thing well and you couldn't depend on the necessary small tools being available. Thus, each compression tool also included its own file spanning system.

https://en.wikipedia.org/wiki/File_spanning

Re:Hackers now use ZIP file concatenation to evade detection by rakoo on Saturday November 16, 2024 @02:34PM (#42158607)
The problem seems to be that each individual split part is valid in itself. This means that the entire file, with the central directory at the end, can diverge from each entry. This is the original issue.
Re:Hackers now use ZIP file concatenation to evade detection by murderfs on Saturday November 16, 2024 @07:19AM (#42156045)
Why do you believe that archiving and compressing belong in the same layer more than sharding does? The unixy tool isn't zip, it's tar | gzip.
Re:Hackers now use ZIP file concatenation to evade detection by edflsafoiewq on Saturday November 16, 2024 @08:11AM (#42156233)
tar|gzip does not allow random access to files. You have to decompress the whole tarball (up to the file you want).
Re:Hackers now use ZIP file concatenation to evade detection by jonjonsonjr on Saturday November 16, 2024 @11:30AM (#42157215)
Even worse, in the general case, you should really decompress the whole tarball up to the end because the traditional mechanism for efficiently overwriting a file in a tarball is to append another copy of it to the end. (This is similar to why you should only trust the central directory for zip files.)
Re:Hackers now use ZIP file concatenation to evade detection by chrisweekly on Saturday November 16, 2024 @01:29PM (#42158105)
I agree!

Also, I enjoyed your Freudian slip:

single respectability tools

->

single responsibility tools

Re:Hackers now use ZIP file concatenation to evade detection by thrdbndndn on Saturday November 16, 2024 @06:45AM (#42155913)
> or split the file

Wait, I'm confused. Isn't this what OP was talking about?

Re:Hackers now use ZIP file concatenation to evade detection by MereInterest on Saturday November 16, 2024 @07:06AM (#42156000)
Splitting the file doesn’t need to be part of the file format itself. I could split a file into N parts, then concatenate the parts together at a later time, regardless of what is actually in the file.

The OP was saying that zip files can specify their own special type of splitting, done within the format itself, rather than operating on the raw bytes of a saved file.

Re:Hackers now use ZIP file concatenation to evade detection by notimetorelax on Saturday November 16, 2024 @10:04AM (#42156779)
Why would you use manual tools to achieve what ZIP archive can give you out of the box? E.g. if you do this manually you’d need to worry about file checksum to ensure you put it together correctly.
Re:Hackers now use ZIP file concatenation to evade detection by rakoo on Saturday November 16, 2024 @02:30PM (#42158587)
Because, as said before, zip managing splits ends with two sources of truth in the file format that can differ while the whole file still being valid
Re:Hackers now use ZIP file concatenation to evade detection by hombre_fatal on Saturday November 16, 2024 @11:27AM (#42157191)
Without knowing the specifics of what's being talked about, I guess it makes sense that zip did that because the OS doesn't make it easy for the average user to concatenate files, and it would be hard to concatenate 10+ files in the right order. If you have to use a cli then it's not really a solution for most people, nor is it something I want to have to do anyways.

The OS level solution might be a naming convention like "{filename}.{ext}.{n}" like "videos.zip.1" where you right-click it and choose "concatenate {n} files" and turns them into "{filename}.{ext}".

Re:Hackers now use ZIP file concatenation to evade detection by justsomehnguy on Saturday November 16, 2024 @04:16PM (#42159418)
> the OS doesn't make it easy for the average user to concatenate files

Bwah! You are probably thinking too much GUI.

    X301 c:\Users\justsomehnguy>copy /?
    Copies one or more files to another location.

    COPY [/D] [/V] [/N] [/Y | /-Y] [/Z] [/L] [/A | /B ] source [/A | /B]
         [+ source [/A | /B] [+ ...]] [destination [/A | /B]]

    [skipped]

    To append files, specify a single file for destination, but multiple files
    for source (using wildcards or file1+file2+file3 format).
Re:Hackers now use ZIP file concatenation to evade detection by thrdbndndn on Saturday November 16, 2024 @11:00PM (#42161873)
Try to concatenate 1000 files with natural sorting names using `copy`. I did this regularly and I have to write a python script it make it easier.

It's much easier to just right click any of the zip part files and let 7-zip to unzip it, and it will tell me if any part is missing or corrupt.

Re:Hackers now use ZIP file concatenation to evade detection by Twirrim on Saturday November 16, 2024 @03:00PM (#42158804)
> Splitting the file doesn’t need to be part of the file format itself. I could split a file into N parts, then concatenate the parts together at a later time, regardless of what is actually in the file.

I'm inclined to agree with you.

You can see good examples of this with the various multi-part upload APIs used by cloud object storage platforms like S3. There's nothing particularly fancy about it. Each part is individually retry-able, with checksumming of parts and the whole, so you get nice and reliable approaches.

On the *nix side, you can just run split over a file, to the desired size, and you can just cat all the parts together, super simple. It would be simple to have a CLI or full UI tool that would handle the pause between `cat`s as you swapped in and out various media, if we hark back to the zip archive across floppy disks days.

Re:Hackers now use ZIP file concatenation to evade detection by _0xdd on Saturday November 16, 2024 @09:18AM (#42156568)
I've had good luck using tools like piping large files through `mbuffer`[1] such as ZFS snapshots, and it's worked like a charm.

[1] https://man.freebsd.org/cgi/man.cgi?query=mbuffer&sektion=1&...

Re:Hackers now use ZIP file concatenation to evade detection by stouset on Saturday November 16, 2024 @01:47AM (#42154892)
If the point is being able to access some files even if the whole archive isn’t uploaded, why not create 100 separate archives each with a partial set of files?

Or use a protocol that supports resume of partial transmits.

Re:Hackers now use ZIP file concatenation to evade detection by ZoomZoomZoom on Saturday November 16, 2024 @03:57AM (#42155361)
This carries the information that all those files are a pack in an inseparable and immutable way, contrary to encoding that in the archive's name or via some parallel channel.
Re:Hackers now use ZIP file concatenation to evade detection by rawling on Saturday November 16, 2024 @03:47AM (#42155329)
Presumably it compresses better if it's all one archive?
Re:Hackers now use ZIP file concatenation to evade detection by thrdbndndn on Saturday November 16, 2024 @06:46AM (#42155915)
Because sometimes your files are very large it's not easy to create separate archives with (roughly) even size.

A single video can easily be over 20GB, for example.

Re:Hackers now use ZIP file concatenation to evade detection by anthk on Saturday November 16, 2024 @10:35AM (#42156914)
nncp, bittorrent...
Re:Hackers now use ZIP file concatenation to evade detection by canucker2016 on Friday November 15, 2024 @10:51PM (#42153990)
I don't think you understand the reason for the ZIP archive file design.

Back in the late 1980s, backup media for consumers was limited to mostly floppy disks, some users had tape/another hard disk.

Say you had a variable number of files to compress and write out to a ZIP archive.

IF you write out the central directory first, followed by all the individually possibly compressed and/or encrypted files, you'd have to calculate all the files to be archived, process them (compress and/or encrypt), write them out, then go back and update the info for the actual compressed values and offsets for the ZIP local entries.

Now if you wanted to add files to the ZIP archive, the central directory will grow and push the following individual compressed/encrypted files further out and you'll have to update ALL the central directory entries since each entry includes an offset from the beginning of the disk - if the archive does not span multiple disks, this offset is from the start of the ZIP archive file.

So that's one reason for why the ZIP central directory is placed at the end of the ZIP archive file. If you're streaming the output from a ZIP program, then placing the ZIP central dir at the start of the file is a non-starter since you can't rewind a stream to update the ZIP central directory entries.

Why do some programs ignore the ZIP central directory as the ONE source of truth?

Before SSDs and their minimal seek latency, coders discovered that scanning the ZIP local entries to be a faster way to build up the ZIP archive entries, otherwise you're forced to seek all the way to the end of a ZIP archive and work backwards to locate the central directory and proceed accordingly.

If the central directory in the ZIP archive is corrupted or missing, the user could still recover the data for the individual files (if all the ZIP local entries are intact). In this case, ignoring the ZIP central dir and scanning sequentially for ZIP local entries is REQUIRED.

The fault here is the security scanners. There's never been any guarantee that the ONLY data in the ZIP archive was only valid ZIP local file entries followed by the ZIP central directory. Between ZIP local file entries, one can place any data. Unzip programs don't care.

Re:Hackers now use ZIP file concatenation to evade detection by Spivak on Saturday November 16, 2024 @12:31AM (#42154566)
The more general principle is that single source of truth is not ideal for data storage where you're worried about corruption. There's a backup MBR on your hard disk at the end, your ext4 filesystem has many backups of your superblock.

When it comes to user data the natural programmer instinct for "is exactly what I expect or fail" which is typically good design, falls to pragmatism where try your hardest to not lose data, partial results are better then nothing, is desired.

Re:Hackers now use ZIP file concatenation to evade detection by tjoff on Sunday November 17, 2024 @04:34AM (#42163068)
Having a backup copy isn't quite the same thing though. It is just a copy of the single source of truth. Not a different implementation or used for a different use case. Also trivial to verify.
Re:Hackers now use ZIP file concatenation to evade detection by unoti on Saturday November 16, 2024 @11:21AM (#42157148)
> coders discovered that scanning the ZIP local entries to be a faster way to build up the ZIP archive entries, otherwise you're forced to seek all the way to the end of a ZIP archive and work backwards to locate the central directory

Would this have worked? Reserve a few bytes at the beginning of the archive at a fixed location offset from the start, and say "this is where we will write the offset to where the central directory will start." Then build the whole archive, writing the central directory at the end. Then seek back to that known offset at the start of the file and write the offset to the central directory. When creating the archive, we can write the central directory to a temp file, and then append that in to the end of the file we're building at the end, and fix up the offset.

Seems like this strategy would enable us to both have a number of files in the archive that are known at the beginning, and also allow us to do a high-speed seek to the central directory when reading the archive.

I imagine people thought about this idea and didn't do it for one reason or another. I can imagine why we didn't do that for Unix TAR-- most tape devices are a one-way write stream and don't have random access. But ZIP was designed for disks; I'm curious why this idea wouldn't have solved both problems.

Re:Hackers now use ZIP file concatenation to evade detection by canucker2016 on Saturday November 16, 2024 @12:12PM (#42157492)
You forgot about the streaming case. ZIP creators can stream the archive out and never seek back earlier in the stream (send the data to another tool or a write-only medium).

The central directory at the end of the archive fits that constraint. Any design where a placeholder needs to be updated later won't.

Re:Hackers now use ZIP file concatenation to evade detection by poincaredisk on Saturday November 16, 2024 @07:32AM (#42156083)
>there are two mechanisms to identify the contents, the per-file header and the central directory

There is only one right, standard mandated, way to identify the contents (central directory). For one or another reason many implementations ignore it, but I don't think it's fair to say that the zip format in ambiguous.

Re:Hackers now use ZIP file concatenation to evade detection by immibis on Saturday November 16, 2024 @11:07AM (#42157052)
Sometimes you want to read the file front-to-back in a streaming fashion.
Re:Hackers now use ZIP file concatenation to evade detection by cxr on Sunday November 17, 2024 @12:45PM (#42165570)
That doesn't change anything wrt what the parent commenter said.

Imagine—

Officer: The reason why I pulled you over is that you were doing 45, but this is a 25 mph school zone right now, and even aside from that the posted speed when this is not a school zone is only 35. So you shouldn't be going faster than that, like you were just now.

Motorist: But sometimes you want to go faster than that.