I didn't say anything about existential risk, and I'm going to assume you meant LLM since training a NN to solve sudoku puzzles has been something you could do as an into to ML project going years back: https://arxiv.org/abs/1711.08028

To me the existential risks are pretty boring and current LLMs are already capable of them: train on some biased data, people embed LLMs in a ton of places, the result is spreading bias in a black box where introspection is significantly harder.

In some ways it mirrors the original stochastic parrot warning, except "parrot" is a much significantly less loaded term in this context.