I agree, LLM are interesting to me only to the extent that they are doing more than memorisation.

They do seem to do generalisation, to at least some degree.

If it was literal memorisation, we do literally have internet search already.