eliomattia

The article can be summarized as: context richer prompt history yields answers that are better aligned with expectations.

> the practical constraint of a finite context window coupled with meta-in-context learning’s rapid prompt length increase

The model is not influenced by the added context. The answers are. The abstractions, or, what the LLM "knows", are in the model itself, whereas the answers are just byproducts.

There is an ongoing related discussion on LLMs lacking world model. One could say LLMs do implement a computable model, albeit a dead, static, non-editable one.

By human feedback in T, I meant RLHF. By chat histories H in T, I meant later feedback from users. Whereas plug-ins and added context can be visualized as g(f(H)), fine tuning could be thought of as g(f)(H). There are humans in the loop, however which loop? Not the one involving the hardware-like f. Unless we retrain de novo.

> This is true for almost everything politicians promote as a "solution"

:-) In their defense, this time they may not even be aware.

> So for most people, they actually see g which is something like g(f(H)))

It is interesting to consider the entailment structure of g(f(H)). f(H), or A, is entailed from H by f. g(f(H)), let us call it A', is entailed from f(H) by g. Using |- as the symbol for entailment, or "entails":

  f |- f(H)
  g |- g(f(H))

All included plugins, embeddings, other API endpoints in g will definitely affect the results, turning answer A, or f(H), into a modified answer A', or g(f(H)). However, what entails f? Nothing in the context of applying g.

To a first approximation, design D and training T are the ones that entail f. T includes all human feedback (training pairs, not direct edits on the entailment in f), and also previous chat histories H for subsequent releases f', however this inclusion is behind decision hierarchies, and the effects of the inclusion on f are largely outside of users' or researchers' precise or accurate control. Training pairs can be added to fine tune, e.g., to filter out violent content, but the results can only be checked downstream of the new f', and fine tuning will retain previous entailments hence acting similarly to a g().

  (D,T) |- f

Let us consider programming, where software S entails output o. What [subject] entails software [direct object]? Programmers P, directly and reliably. Not only can programs be affected precisely and accurately. If we want to modify a program, we do not wrap it inside another program g, hoping that the functional composition will yield the desired result, which will invariably contain entailment from the one we are stuck with. Instead, we edit it.

  S |- o
  P |- S
  S |- P

Programming is interactive, between programmers and software, whereas with LLMs users are downstream when it comes to wanting to affect directly the entailment patterns in f. The human-like chat UX of LLMs leads us to think that chatting with them, giving them human feedback, prompts, or guidelines, or adding plugins and context will naturally flow into us being able to affect f, and that since humans can be manipulated, then so can LLMs. The core of the matter is that f, which entails issues, cannot be affected easily, precisely, or accurately, despite any g that we may want to wrap around it.

There are two divergent analogies to analyze. Compared to programming software, LLMs are experientially closer to the malleability of interacting with humans on the one hand, while functionally closer with hardware entailment-wise, on the other hand. Using symbols that do not overlap with the above: in a computer, hardware W entails that the execution of software S can yield output o from input i. Programmers have the freedom to edit S and experiment with their i/o data. However, it is W, unentailed internally, that defines the boundaries of what is possible for a user: changing W requires hardware engineers and new designs. If W is a desktop computer, programming car engine simulations as S will not turn W into a car. This creative example is meant to indicate that the causation cycle behind a functional component such as W that is unentailed internally at the user/computer level is inherently layered. Similarly, the interactions f has had with H might fully, minimally, or might not at all be considered in the next release. In the case of computers, a user cannot modify W, but hardware engineers can, with full control. With LLMs, not even the researchers can fully control f, for example, to impose that violent themes not be discussed, even using impersonations or a developer mode.

The reason it is important to remain aware that f is not necessarily coevolving with all provided H is the social ease of overlooking how each component is entailed in the current mainstream paradigm. In LLMs, f literally remains unchanged with each interaction, yet a common impression is that we can affect LLMs by chatting with them since the ChatGPT UX is strikingly similar to the experience of chatting with a human. It feels plausible that the effect of just talking to LLMs will be as strong, or even stronger than editing code because when talking to an intelligent human, H can indeed affect their biological f. However, the analogy that holds the strongest with LLMs is that the entailment of f is close to that of hardware W: changing f is much more akin to requesting hardware engineers for edits to W, with the caveat of giving up on editing precision or accuracy, especially if attempting to mask or remove harmful information and when not retraining from scratch. It is true that feedback from H will affect future releases f', but 1) in each release, f remains immutable just like W throughout the interactions, 2) feedback is integrated with delays and slowly, 3) editing is not generally available to users directly except for more superficial fine-tuning, 4) feedback is orders of magnitude less impactful compared to training corpora and design decisions in foundation models, and 5) unlike with designing actual hardware W, even by calling f ChatGPT as a whole, including its many releases, changing f is neither precise nor accurate, as one would imagine an editing process to be and as modifying software S directly is.

Returning to the jailbreak article and using the parallel with the hardware analogy, I assume the intention of the legislators is ultimately to change f, the source of the causation that entails all answers A in chat history H, both by further fine-tuning the models and by adding a list of chat guidelines that are fed as part of the prompt. However, due to the entailment hierarchy of the general architecture, the latter attempt will have zero impact on f itself, thereby not addressing the issue at a fundamental level, while the former strategy will only have a limited effect, due to a fundamental lack of direct control on f: researchers do not have a way to precisely and accurately steer f, and users are even further separated from affecting it.

This is not meant to be dismissive of the current achievements, the results are impressive and the techniques are steadily improving, but rather a critical look at the entailment structures at hand, both perceived and observed, and at the available strategies regarding safeguards. I find it interesting to ask: how can f be made functionally closer to programming S than to W?

There are assumptions here that are intimately related to the meta questions mentioned.

> Prompt injection or prompt attacks are well known and likely impossible to guard against.

They are impossible to guard against under the assumption that the current LLM paradigm is all there is and all there could possibly be. There could be other realizations of AI. The latest impressive achievements are yielding an ongoing identification of the current computational approaches with human intelligence itself, with how we humans model reality by using natural language, and with how we could ever imagine that a computer can model reality via natural language. These are all strong assumptions and are very common even amongst researchers.

> Can you really get a human to be invulnerable to manipulation?

Most definitely not, but we are specifically not talking about humans, but about:

> AI, which relies upon computers, algorithms, and numbers written on memory

I am not making a case for machines being completely invulnerable to manipulation, which requires an analysis of the entailment structures of reality that would yield that complete invulnerability is impossible, but for better direct control on the internals, rather then relying on external instructions that easily undergo jailbreaks with simple prompt attacks.

> Why would we expect the machines to be any better?

One argument is: because they can be programmed and the memory they rely upon for its algorithms can be edited directly, with both accuracy and precision. The missing piece is how to model reality via natural language in a computer, in a way that we would know what to edit in order to affect the model with accuracy and precision.

LLMs, currently, are non-editable. When interacting with ChatGPT, its answers A will be generated from chat history H, which includes prompts and guidelines, by an immutable function, or program, f: A = f(H). It is remarkable that, in LLMs, f cannot be edited and is never entailed by the individual chat H. Since we can have multiple exchanges in the same history, H will itself contain information (entailment) from f, but never the other way around: f is not entailed by A or H, it is fixed, and only entailed externally by the design and training steps. f can be fine-tuned, yet it will retain remnants of past training, hence it cannot be truly be edited at will unless we retrain the whole model. Even then, control on f is neither accurate nor precise.

It seems that non-editable LLMs remove some of the agency that is inherent in programming: editing the internals of a program to shape the entailment structures that we want to realize, with accuracy and precision.

I am by no means indicating that editable AI models that can be steered are easy to achieve, rather that the very possibility thereof is rarely mentioned, and often implicitly assumed not to exist in absolute statements that in fact only strictly apply to the current mainstream approaches.

As a programmer, I find it fascinating to build things from the ground up, with the inner workings either in full display or readily accessible for editing. With AI, the need to beg it to please behave, with a long list of things to do and not to do and a resounding order not to disclose such a list, is becoming commonplace.

Obviously, finding jailbreaks in LLMs is extremely important and consequential. However, there are meta questions around modern AI that remain valid, and this article is a reminder: is a continuous and direct feedback loop between code and coder a thing of the past? To what extent should we accept that LLMs are trained one-way, that we can only truly edit them with expensive trial-and-error retraining runs, hence, all we are left with is asking kindly? Are the current implementations all, or are we dealing with just one possible paradigm? Do we want AI, which relies upon computers, algorithms, and numbers written on memory, to be fundamentally programmable?

Really interesting, also diff-based, and 3.5 years in development.

On the homepage I read "An in-memory, distributed, and open-source document graph database". Do you know whether the whole database, including all documents, needs to be in memory, and what happens when the datasets exceed available memory space? Or is it perhaps in-memory per document, one-by-one?

Do you have to create diffs manually with terminusdb (CLI example below from https://terminusdb.com/products/terminusdb/), or can they be detected automatically from, e.g., SQL database tables or files in a folder, similarly to Git monitoring a working directory and committing changes based on its contents?

  # Add more philosophers to new branch
  echo '{ "name": "Plato" }' | terminusdb doc insert admin/philosophers/local/branch/changes
  echo '{ "name": "Aristotle" }' | terminusdb doc insert admin/philosophers/local/branch/changes

  # Look at the difference between branches
  terminusdb diff admin/philosophers --before-commit main --after-commit changes | jq

  # Apply the differences to main
  terminusdb apply admin/philosophers --before-commit main --after-commit changes

I just found this blog post while doing research. It seems Palantir Foundry, which does not come up often when researching git for data tools, includes a version control system for datasets that stores diffs in their own cloud-based filesystem. According to the author, one of the founding engineers of the platform, diffs are:

> particularly useful for append-only datasets of immutable records such as system logs or sensor readings which are often among the largest (and fastest-growing) datasets our customers use

Diffs seem to consist of additional files in separate folders:

> behind the scenes we effectively store each diff in a separate folder in the backing file system (e.g.,datasetA/diff1, datasetA/diff2, …) so that the whole dataset is simply represented by datasetA/*.

Without exposing technicalities, the author suggests that the delete use case is taken care of logically and not physically, since datasetA/* may not reflect the actual whole dataset. I infer that they might be logging changes under the hood in a Git-like fashion.

> It’s a bit more complicated than this because users can selectively delete files from those diffs

However, it seems that the versioning raw data they manage are not available to clients or users directly:

> a simple request that we frequently get from our customers: “can we export our datasets from Palantir Foundry to our existing data lake or S3 bucket?“ While this is of course possible, it is important to understand that such exported datasets lack precisely those versioning and sandboxing features that make Foundry a great tool for collaborative data engineering.

This could be a mechanism for vendor lock-in, tied to the very important ACID guarantees of their implementation.

Some background on why I came across their post, here: https://news.ycombinator.com/item?id=35930895

Fully agree. Compression in many cases removes the ability to diff easily, however. In a large dataset where, in terms of size, 1% of the original data undergoes changes, or new data the size of 1% of the original dataset is added, I think compressing does not compare with just deduplicating the unchanged 99% in terms of storage, but when speed is the #1 factor, the discussion is more nuanced. It might be interesting to have a combination of deduplication and better compression of the changes, in some form, to get the optimal tradeoff. Repo sizes in ML these days are high, I'm curious which repository compression techniques are being evaluated and deployed.

*I have just (only now) read the second paragraph in your message. Not sure if that came across correctly, that first sentence was too compressed.

The two are not mutually exclusive, in principle. Depending on workflows, sizes, and change frequencies, each has advantages. Sparse checkouts can be useful with small files that specific individuals can focus upon, among other things. This is for larger datasets that change daily and for facilitating versioning in workflows with incoming data. This is an interesting use case: https://news.ycombinator.com/item?id=35763004

Just read the second paragraph. Currently expanding merge resolution assistance to deal with the general merge conflict case, as well as implementing revert and cherry-pick assistance. Unsure if that is what you were wondering? You probably don't want to do that with 100 GB if most of your commits are new data, rather than changes, yet I wonder whether all incoming new files are then queued into the same pipelines and the reason why they are separate files is not to have to deal with one giant cumulative file for which the older parts would not be deduped in Git LFS, which is a great reason, or whether those files are anyway different file types in terms of contents and intended use, another great reason, or something else altogether. Are you processing data by including anything in a given folder in a given commit?

Sorry I should have been more specific, I meant block deduplication, or any form of deduplication at a level lower than the entire file. File deduplication can only get you so far, depending on the use case. XetHub does block deduplication, whereas I am implementing data-level deduplication, which is slower in recreating dataset snapshots (can be parallelized and delegated), but allows savings on disk space with small but frequent changes and can be tied to collaborative features to show diffs, comment on them, and revert or edit changes where needed, all while pointing clearly to specific commits. And also potentially fork data or cumulative changes.

Yes I meant either checking out other branches locally, or in the general case pointing to another branch to indicate to any services to make data from that branch available to wherever it's consumed. I am assuming that each incoming new file is then added to data pipelines, possibly just a few. Sounds like you are in the sweet spot where you have the speed you want and, given unfrequent changes, you are fine with the versions taking up terabytes on Azure, since they are mostly new data.

That is really interesting and begs the question of how frequently you have changes in your data that lead to new commits. I am assuming here that you don't dedupe anything, that is, you throw the entire files into Azure with each version, since it's cheap enough for your purposes. Also, how frequently do you move head, even without committing anything new, perhaps to use another branch?

Git does do compression on repos, but the fact that versioning repositories with (huge) data is still an open problem suggests that it is not the kind that fixes it. I might be mistaken, are you aware of any interesting compression methods applied to version control?

Which repo size after the filters do you work with on your machine and how many GBs do you have in Git LFS, that is, in the cloud? I hear people complain about costs, but it depends upon scale and change frequency, which can increase total repo size.

It's like GVFS, but for pieces of a file at a time as well: rows, columns, or cells. In git, you can see in the video the same file diff being overwritten, representing a new piece in every commit. A snapshot is recreated by putting those pieces together. Since S3 pointers are used, it is neither partial nor shallow clone strictly, as all the objects and trees are downloaded in the current implementation, but the pointers enable similar delaying behavior, like with DVC.

Sorry if the repo size is unclear, hope this is better: ~180 kB: current state at the branch head, includes pointers to S3 (exact size depends upon packs and indices), plus full history, also with pointers ~890 MB: current state at the branch head, after downloading all files referenced by pointers in the Git history from S3, plus full history, with pointers ~130 GB: commit history, this is what the repo would weigh on DVC or Git LFS

With increasing repo size (even the 2nd, 890 MB state), this enables working on the 1st (kBs) and still commit changes.

Full checkouts of large data repositories are problematic. In the video, a workflow that does not require full checkouts of the datasets and still allows to commit diff-based changes in Git. This naturally applies to new data, and can be applied to edits or deletions provided the old data is known - this is to ensure the creation of bidirectional diffs that enable navigating Git history both forward and backward, useful when caching snapshots. Feedback is most welcome!

On open-source: should this toolset be open sourced? Which license to choose?

Full checkouts of large data repositories are problematic. In the video I present a workflow that does not require full checkouts of the datasets and still allows to commit diff-based changes in Git. This naturally applies to new data, and can be applied to edits or deletions provided the old data is known - this is to ensure the creation of bidirectional diffs that enable navigating Git history both forward and backward, useful when caching snapshots. Feedback is most welcome!

On open-source: should this toolset be open sourced? Which license to choose?

I am working on a solution on top of Git, but storing diffs only, can integrate with MySQL and S3, can create version snapshots. You would have the videos in a bucket and the version-controlled history with links to the videos in another, also on S3. Data can be added as a diff commit and later merged into production datasets. You would own the history via a readable Git repo and pointers to your versioning S3, on top of any snapshots. No UI yet, though. Many open questions, some of them: - Data ingestion: how often per person, how many new videos each time, how many field personnel? - Dataset carveouts: how often, based on what exactly would you filter? - Metadata: which ones per video, how often querying, on a specific version of the datasets? A few query examples would help to imagine where the metadata should live. My email is in the profile, feel free to reach out, most likely my solution is too early stage for your needs.

Important points. I aim for version control for data repositories with HDD efficiency, visualization of diffs for collaboration, and API accessibility of individual datasets from multiple identifiable versions from git. Datasets can then be streamed into a database individually. Migrating by processing diffs from commits, a future possibility. Fast direct database-like repository access with in-situ editing is not my initial goal, rather a clonable Git repository that is separate from the database working areas, can be connected to MySQL using i/o pipelines, and can easily export datasets versions as labeled snapshots.

On diffs: diffing workload is lazy and needed just once upon requesting a commit for a repo, the bidirectional diff results (incl. index and columns changes) - not the newly changed files - are then committed as csv or pointer objects, git natively supports seeing such objects as new. I had to write an engine to rebuild and serve snapshots from git history, now on localhost with posting to S3, later also in the cloud. Diffing a dataset (csv, Excel, SQL) against the current checked out version, which now resides in a gitignored "datasets" folder in the working directory, now takes ~20 seconds on a 1 GB CSV dataset with 10M rows. Diffing is not always needed, can be bypassed with incremental workflows (new data daily), committing just the diff. I can handle repos of 10s of GBs, with individual datasets of GBs each. Where to put the compute workload of diffing, checking out, and building snapshots is under careful consideration.

Merge will be assisted, with files in S3 or without: 3-way comparing commits boils down to reusing features from the snapshot rebuild engine, starting from the common ancestor and using only the diffs, and handling conflicts. Merging small changes on large datasets involves dealing with the small changes only. Data and diffs can come from S3 if not already on localhost, to merge data, not pointers. Visually presenting diffs in non-adjacent commits requires UI, current git tools would not interpret history correctly.

dolt came up when searching git for data, it seems great, though I have never used it. I know it works on prolly trees rather than on top of git. I am really curious to learn about that choice, why exactly not on git? How can you offer data removal from history without rebuilding repos? Especially here in the EU, an ongoing conversation is taking place about structured data documentation, collaboration, and tech design choices that affect privacy.

I have built version control for data, on top of git itself, that can commit and push incremental diffs. By tagging in git, a version snapshot can be created. S3 can be configured (a) for heavy files and diffs referenced by pointer objects and (b) for shareable snapshots, identified by repo, tag name and commit sha. The diffs operate at row, column, and cell level, not by block deduping. Datasets must have some tabular structure. The data will go wherever pushed, and to user S3 if configured.

The burden of checking out and building snapshots from diff history is now borne by localhost, but that may change. Smart navigation of git history from the nearest available snapshots, building snapshots with Spark, and other ways to save on data transfer and compute are all on the table. Merge conflict resolution is in the works. This paradigm enables hibernating or cleaning up history on S3 for datasets no longer necessary to create snapshots, like those that are git removed if snapshots of earlier commits are not needed. Individual data entries could also be removed for GDPR compliance using versioning on S3 objects, orthogonal to git.

The prototype already cures the pain point I built it for: it was impossible to (1) uniquely identify and (2) make available behind an API multiple versions of a collection of datasets and config parameters, (3) without overburdening HDDs due to small, but frequent changes to any of the datasets in the repo and (4) while being able to see the diffs in git for each commit in order to enable collaborative discussions and reverting or further editing if necessary. Some background: I am building natural language AI algorithms that (+) operate on editable training datasets, meaning changes or deletions in the training data are reflected fast, without traces of past training and without retraining the entire language model (I know this sounds impossible), and (++) explain decisions back to individual training data. LLMs have fixed training datasets, whereas editable datasets call for a collaborative system to manage data efficiently.

I am open to everything including thoughts, suggestions, constructive criticism, and use case ideas.