This is true for almost everything politicians promote as a "solution"
> f literally remains unchanged with each interaction
For now, on a model weights level, sure. But we have no insights into the random seed, chain of calls, or external inputs that may be included. f may be thought of to include much more than "just the model", even today. The ChatGPT is making multiple API calls, one of which is the moderation endpoint after the completion call on the client side. So for most people, they actually see g which is something like g(f(H)))
For example, ChatGPT plugins change the game, as do the embedding databases and other attempts to deal with limited context.
:-) In their defense, this time they may not even be aware.
> So for most people, they actually see g which is something like g(f(H)))
It is interesting to consider the entailment structure of g(f(H)). f(H), or A, is entailed from H by f. g(f(H)), let us call it A', is entailed from f(H) by g. Using |- as the symbol for entailment, or "entails":
f |- f(H)
g |- g(f(H))
To a first approximation, design D and training T are the ones that entail f. T includes all human feedback (training pairs, not direct edits on the entailment in f), and also previous chat histories H for subsequent releases f', however this inclusion is behind decision hierarchies, and the effects of the inclusion on f are largely outside of users' or researchers' precise or accurate control. Training pairs can be added to fine tune, e.g., to filter out violent content, but the results can only be checked downstream of the new f', and fine tuning will retain previous entailments hence acting similarly to a g().
(D,T) |- f
S |- o
P |- S
S |- P
RLHF is Reinforcement Learning from Human Feedback
Thus you cannot get ChatGPT without humans in the loop, making it quite sensitive and irreproducible.
Your premise is that these are not editable models, but you restrict that to a situation that is not true in practice. These models are constantly changing based on human feedback, updated information, config changes, and weight manipulation.
If you follow the conversation around LLMs, one of the top issues people have is inconsistent responses and the models changing under their prompts and breaking them.
> We find that meta-in-context learning adaptively modifies priors over latent variables, ultimately leading to priors that closely resemble the true statistics of the environment. Furthermore, our analysis reveals that meta-in-context learning can not only be used to change prior expectations but is also capable of reshaping an LLM’s learning strategies
This was done using OpenAI's public facing APIs
Even if it is not a permanent "edit" it still influences the model at the lowest levels
> the practical constraint of a finite context window coupled with meta-in-context learning’s rapid prompt length increase
The model is not influenced by the added context. The answers are. The abstractions, or, what the LLM "knows", are in the model itself, whereas the answers are just byproducts.
There is an ongoing related discussion on LLMs lacking world model. One could say LLMs do implement a computable model, albeit a dead, static, non-editable one.
The reason it is important to remain aware that f is not necessarily coevolving with all provided H is the social ease of overlooking how each component is entailed in the current mainstream paradigm. In LLMs, f literally remains unchanged with each interaction, yet a common impression is that we can affect LLMs by chatting with them since the ChatGPT UX is strikingly similar to the experience of chatting with a human. It feels plausible that the effect of just talking to LLMs will be as strong, or even stronger than editing code because when talking to an intelligent human, H can indeed affect their biological f. However, the analogy that holds the strongest with LLMs is that the entailment of f is close to that of hardware W: changing f is much more akin to requesting hardware engineers for edits to W, with the caveat of giving up on editing precision or accuracy, especially if attempting to mask or remove harmful information and when not retraining from scratch. It is true that feedback from H will affect future releases f', but 1) in each release, f remains immutable just like W throughout the interactions, 2) feedback is integrated with delays and slowly, 3) editing is not generally available to users directly except for more superficial fine-tuning, 4) feedback is orders of magnitude less impactful compared to training corpora and design decisions in foundation models, and 5) unlike with designing actual hardware W, even by calling f ChatGPT as a whole, including its many releases, changing f is neither precise nor accurate, as one would imagine an editing process to be and as modifying software S directly is.
Returning to the jailbreak article and using the parallel with the hardware analogy, I assume the intention of the legislators is ultimately to change f, the source of the causation that entails all answers A in chat history H, both by further fine-tuning the models and by adding a list of chat guidelines that are fed as part of the prompt. However, due to the entailment hierarchy of the general architecture, the latter attempt will have zero impact on f itself, thereby not addressing the issue at a fundamental level, while the former strategy will only have a limited effect, due to a fundamental lack of direct control on f: researchers do not have a way to precisely and accurately steer f, and users are even further separated from affecting it.
This is not meant to be dismissive of the current achievements, the results are impressive and the techniques are steadily improving, but rather a critical look at the entailment structures at hand, both perceived and observed, and at the available strategies regarding safeguards. I find it interesting to ask: how can f be made functionally closer to programming S than to W?