It's not clear at all whether the model is a derivative work from a copyright standpoint. It's definitely not settled, the law isn't very explicit and as far as I know, there is no reasonable precedent yet - and arguably that would be one of the key issues decided (and set as precedent) in these first court battles.

First, there is a reasonable argument that the model weights may be not copyrightable at all - it doesn't really fit the criteria of what copyright law protects, no creativity was used in making them, etc, in which case it can't be a derivative work and is effectively outside the scope of copyright law. Second, there is a reasonable argument that the model is a collection of facts about copyrighted works, equivalent to early (pre-computer) statistical ngram language models of copyrighted books used in e.g. lexicography - for which we have solid old legal precedent that creating such models are not derivative works (again, as a collection of facts isn't copyrightable) and thus can be done against the wishes of the authors.

Fair use criteria comes into play as conditions when it is permissible to violate the exclusive rights of the authors. However, if the model is not legally considered a derivative work according to copyright law criteria, then fair use conditions don't matter because in that case copyright law does not assert that making them is somehow restricted.

Note that in this case the resulting image might still be considered derivative work of an original image, even if the "tool-in-the-middle" is not derivative work.