A trained LLM is software the same way an opaque binary is software. To be truly 'soft' you need to be able to recreate or change it, i.e. have the 'source' in this case the means to recreate/train it.