Mostly agree, but:

> So the answer to "why not start now" is "because it seems like it will be a waste of time".

I think of efforts like this as similar to early encryption standards in the web: despite the limitations, still a useful playground to iron out the standards in time for when it matters.

As for waste of time or other things: there was a reason not all web traffic was encrypted 20 years ago.

Agree with most of your points, but a LargeLM, or a SmallLM for that matter, to construct a simple SQL query and put it in a database, they get it right many times already. GPT gets it right most of the time.

Then as a verification step, you ask one more model, not the same one, "what information got inserted the last hour in the database?" Chances of one model to hallucinate and say it put the information in the database, and the other model to hallucinate again with the correct information, are pretty slim.

Many sources of information contain inaccuracies, either known at the time of publication or learned afterward.

Education involves doing some fact checking and critical thinking. Regardless of the strength of the original source.

It seems like using LLMs in any serious way will require a variety of techniques to mitigate their new, unique reasons for being unreliable.

Perhaps a “chain of model provenance” becomes an important one of these.

If you already know that your model contains falsehoods, what is gained by having a chain of provenance? It can't possibly make you trust it more.

People contain a shitload of falsehoods, including you, yet you assign varying amounts of trust to those individuals.

A chain of providence isn't much different then that person having a diploma, a company work badge, and state issued ID. You at least know they aren't some random off the street.

That only provides value if you have knowledge of how the diploma-issuing organisation is working.

If not, it's just a diploma from some random organisation off the street.

So, wake me up when you know how OpenAI and Google are cooking their models.

Per https://en.wikipedia.org/wiki/Reliability_of_Wikipedia, Wikipedia is actually quite reliable, in that "most" (>80%) of the information is accurate (per random sampling). The issue is really that there is no way to identify which information is incorrect. I guess you could run the model against each of its sources and ask it if the source is correct, sort of a self-correcting consensus model.

I'm generally pretty pro-Wikipedia and tend to think a lot of the concerns (at least on the English version) are somewhat overblown, but citing it as a source on its own reliability is just a bit too much even for me. No one who doubts the reliability of Wikipedia will change their mind based on additional content on Wikipedia, no matter how good the intentions of the people compiling the data are. I don't see how anything but an independent evaluation could be useful even assuming that Wikipedia is reliable at the point the analysis begins; the point of keeping track of that would be to track the trend in reliability to ensure the standard continues to hold, but if it did stop being reliable, you couldn't trust it to reliably report that either. I think there's value in presenting a list of claims (e.g. "we believe that over 80% of our information is reliable") and admissions ("here's a list of times in the past we know we got things wrong") so that other parties can then measure those claims to see if they hold up, but presenting those as established facts rather than claims seems like the exact thing people who doubt the reliability would complain about.

Here’s an actual reliable source on the reliability of Wikipedia that confirms the meta Wikipedia article: https://amp.smh.com.au/national/evidence-suggests-wikipedia-... https://sci-hub.ee/https:/asistdl.onlinelibrary.wiley.com/do...

Wikipedia may be reliable, but you should never cite anything on its own reliability lmao

IME 99% somebody is feigning concern about Wikipedia's "reliability" it's because they want to use sources that are far more suspicious and unreliable.

> "most" (>80%) of the information is accurate (per random sampling)

Disinformation isn't random though; there's not an equal chance that information is misleading on ever topic.

Most information can be accurate while still containing dangerous amounts of disinformation.

There is a difference between bugs and attacks. I think we are trying to solve attacks here. In an attack, I might build an LLM targeting some service that uses LLMs to execute real world commands. Adding providence to LLMs seems like a reasonable layer of security.

Now we shouldn’t be letting a random blob of binary run commands though right? Well that is exactly what you are doing when you install say Chrome.

A service should not use LLMs to execute real world commands. Ever.

I go back far enough in time and people said the same about Javascript in the browser, yet here we are, and will also be with LLMs.

Undoability is going to be a consideration. We let people use credit cards with practically no security for convenience, because the cost of reversing a few transactions of refunding people for fraud is low enough.