The scary applications are the ones where it's not so easy to check correctness...

As for things that can actually be incorrect, that has always been impossible, but we accept the human consensus to be a close enough approximation. With that, verifying 'correctness' to the degree that is possible is actually quite easy through validating it across many different LLMs trained on the human consensus. They will not all hallucinate identically. If convergence is found, then you have also found the human consensus. That doesn't prove correctness — we have never had a way to do that — but it is equivalent to how we have always dealt with establishing what we believe is correct.

Despite your insistence, I think you will find that the human consensus is that it useful. The human consensus is especially biased in this case, I will grant you that, but it seems few humans wish they were bears in the forest. Our ability to so effectively communicate in such a messy, imperfect environment is what has enabled us to be unlike all the other animals.

It might not sound like it should work on paper, but in the real world it does.

asdjklfh asdjhgflkj bveahrvjkhgv hjagsdfhj hgertjhga ads fhdfjmjhkr

Nope, that's incorrect english.

Turns out that because we've defined "words" as a thing that means a thing, now there are rules around "language" and "words". So while you're welcome to invent whatever combination of sounds you prefer to mean what you like, those sounds can be "correct" or "incorrect" as soon as other people become involved, because now you've entered into a social construct that extends beyond yourself.

So again your conclusion is technically correct, in a navel-gazing "the universe is what I perceive" sort of way, but counterproductive to use as a building block for communication.

There is no correct or incorrect here, but I will say it looks perfectly fine to me. I don't understand it. Is that what you are trying to communicate? There are many words I don't understand; even ones found in the dictionary. That is nothing new.

Here's the magic: I don't need to understand. Nobody is born with the understanding. Where communication is desired, we use other devices to express lack of understanding and keep trying to convey intent until a shared understanding is reached. I don't yet understand what that means, but assuming you are here in good faith, I eventually will as you continue to work to communicate your intent behind it.

I know computer people who spend their days writing in programming languages that never talk back struggle with this concept, but that doesn't change the reality.

this is important, i feel like a lot of people are falling in to the "stop liking what i don't like" way of thinking. Further, there's a million different ways to apply an AI helper in software development. You can adjust your workflow in whatever way works best for you. ..or leave it as is.

Much of the AI antipathy reminds me of Wikipedia in the early-mid 2000s. I remember feeling amazed with it, but also remember a lot of ranting by skeptics about how anyone could put anything on there, and therefore it was unreliable, not to be used, and doomed to fail.

20 years later and everyone understands that Wikipedia may have its shortcomings, and yet it is still the most impressive, useful advancement in human knowledge transfer in a generation.

I think LLMs as a technology are pretty cool, much like crowdsourcing is. We finally have pretty good automatic natural language processing that scales to large corpora. That's big. Also, I think the state of the software industry that is mostly driving the development, deployment, and ownership of this technology is mostly doing uninspired and shitty things with it. I have some hope that better orgs and distributed communities will accomplish some cool and maybe even monumental things with them over time, but right now the field is bleak, not because the technology isn't impressive (although somehow despite how impressive it is it's still being oversold) but because silicon valley is full of rotten institutions with broken incentives, the same ones that brought us social media and subscriptions to software. My hope for the new world a technology will bring about will never rest with corporate aristocracy, but with the more thoughtful institutions and the distributed open source communities that actually build good shit for humanity, time and time again

Garbage in, garbage out still applies.

Which is why we shouldn't be surprised when AI, trained on the collective wisdom of facebook posts and youtube comments, keeping lying to us.