First, licenses matter. Photoshop.exe is closed source first and foremost because the license says so.
Second and more importantly for this discussion, Adobe doesn't prefer to work with hexedit, they prefer to work with the source code.
OpenAI prefers to fine tune their existing models rather than train new ones. They fine tune regularly, and have only trained from scratch four times total, with each of those being a completely new model, not a modification.
That means the weights of an LLM are the preferred form for modification, which meets the GPL's definition of 'source code':
> The “source code” for a work means the preferred form of the work for making modifications to it.
This "preference" is solely based on the cost, not convenience, unlike in the GPL definition. The cost is going to change fast and should not be a part of the definition.
In order for the weights to take all the training data and embed it in the model, by definition, some data must be lost. That data can't be recovered, no matter how much you fine tune the model. Because we can't, we don't know how alignment gets set, or the extent of it.
The closet thing these things have to source code is the source code and training data used to create the model. Because that's what's used to created the model. How big a system is necessary to train it doesn't factor in. It used to take many days to compile the Linux kernel, and many people at the time didn't have access to systems that could even compile it.