It probably flags it as unsafe and triggers smart screen. Meaning you have to go out of your way to use the file.

As for malware that would be unzipped when using an external zip file; first you would need to trigger the zip bomb on defender but not the external tool, and second defender will still scan the individual files getting unziped by that tool.

Maybe do a self extractor that's specially crafted to ignore the zip-bomb part and only extract the malware part? Then the self extractor can execute the malicious code after extracting to memory.

I work at a threat intelligence firm which provides services for aggregating desktop antivirus engines & specialized malware detection tooling; I've submitted dozens of public and manually constructed compression-bombs for analysis (including this one). Many antivirus engines, even very small vendors, handle this case better than you might imagine.

I'm just thinking of strategies for dealing with it. Sandboxed whitebox fuzzing seems like one way of dealing with executables that could take some unknown commands directing the malicious behaviour - but you introduce the possibility of a sandbox breach making the antivirus itself the vector for execution.

The other possibility I see harks back to algorithms - build a DAG from the archive's interfile dependencies and run an iterative deepening search through the structure against some heuristic checking for malicious design.

I've never gotten into anything security related (other than reading Schneier's blog) but the cat and mouse game is fascinating.

In such a case could anything actually extract that malware and run it?

Sure, as long as that other thing knew not to treat the file as a zip, or had a mechanism to skip over or blank out the first chunk of the file.

I wonder how Windows Defender would treat something that looked like a self-extracting archive? Perhaps the archive portion could be this zip bomb affair, but the executable portion had a small change in it to bypass that and do something else nefarious instead, eg hand execution control to a point later in the file.

I don't see why not; just have the payload stored after the bomb - if there are no share dependencies (which there wouldn't be) there's no reason why they shouldn't be independently extractable.