Google's thumbnails are a purely mathematical transformation on images (no copyright themselves), and yet are considered a transformative use.
I believe that trained models are similarly a purely mathematical transformation of {data}, but is transformative in what that can be used for going forward.
"Can" bearing a lot of weight in that sentence.
It's how the human, with agency, uses the model that may be a derivative or copyright infringing use - not the model itself nor necessarily the output.
The output of a generative AI may be similar enough to an existing work that it is derivative of that work. It is possible to construct a prompt that infringes on an existing work even if that work wasn't part of the training data.
For that case, consider you drew a picture. That picture that you just drew isn't part of any training data. I could presumably look at it and describe it with sufficient detail that something similar enough would be generated... and that may be considered a derivative work. The same test could be applied to me describing it to someone on Fiverr with the same outcome.
If I were to publish that work by the generative AI or Fiverr - who would be infringing on copyright? me? or the black box that may be AI or Fiverr that created a picture based on my prompts?
The inputs could be copyrighted and the weights could be copyrighted if creating the weights from the inputs is (legally) regarded as a transformative use. And I think it could reasonably be considered to be transformative - the weights don't look anything like the input data.
Disclaimer: IANAL. So far as I know, no court has ruled on whether this qualifies as a transformative use. I take no position on how the courts will actually rule. I merely say that they could regard this as transformative use. (But see jerf's "creativity" argument for another hurdle that weights must pass to be copyrightable.)