Very good point! I do find it incredible that LLMs can do regression with just in-context learning. This is one of my favorite examples of a task that is mind-blowing to a researcher but a meh or huh to the public.

The main issue is there isn't enough evidence to say one way or another though, I think, which is a complaint of how tasks from modern large models are shared by OpenAI and Google.