https://www.cs.utexas.edu/users/EWD/transcriptions/EWD06xx/E...
The entire field of prompt engineering is doomed from the start. You can't win if the win condition is wasting your time.
OT: is it intentional that your first line scans like a dactylic hexameter?
Yes.
No, not really. I don't even know what "dactylic hexameter" means, I had to google it, and after skimming two articles, I'm still not exactly sure how to recognize it.
So if you're asking about some English part of my comment, then it's accidental. If you mean the Latin bit, then... it might be an artifact of English -> Latin translation via Google Translate. And/or something about the structure of the original "system_prompt.txt" text. Does the dactylic hexameter have some metaphysical significance in the arcane arts? Maybe when it shows in a "prompt hack", it's not by coincidence.
One of my favorite recent projects is called Parsel:
Parsel: A (De-)compositional Framework for Algorithmic Reasoning with Language Models
https://arxiv.org/abs/2212.10561
Here's a notebook with an introduction:
https://github.com/ezelikman/parsel/blob/main/parsel.ipynb
And here's a GUI interface the author has been developing:
http://zelikman.me/parsel/interface.html
I've been working on an augmented large language model that given these few-shot exemplars can build the below fully-functional ToDo App: ==
https://github.com/williamcotton/transynthetical-engine/tree...
https://www.williamcotton.com/articles/junie-browser-builder...
All of this is still very rough around the edges, prone to errors of various kinds, and generally not ready for prime time, but anyone is welcome to play around with what is there!
Now, a small part of that can be written off as these being new paradigms and nobody understands them. But prompt engineering is, in much larger part, completely unlike writing code in a programming language, because it can never be understood "from first principles", because neural networks are inscrutable and stochastic by their very nature.
It's like trying to write production code in an esolang like Malbolge.
Herein lies the problem, though. Either there are patterns to it, which can be discovered, formalized and understood, or there are no patterns to it. If it's the former, sticking to natural language is stupid, for the same reason eyeballing something is stupid, when a mathematical formula will yield you better results for less effort. If it's the latter, sticking to natural language is stupid too, because the whole system is useless - if there are no patterns to study, you may just as well flip a coin or read from /dev/urandom.
Now, the very existence of prompt engineering tells us we're likely dealing with the first case - with understandable patterns. However, our systems are not black boxes. Prompt engineering is, at its best, turning interactions with LLMs into an empirical science, which makes no sense when dealing with human-made artifacts. We don't need to discover the patterns, we can read them off the thing, and we can adjust the thing to manifest different patterns.
> It's like trying to write production code in an esolang like Malbolge.
It's more like trying to learn programming via scientific method: running sets of random characters through the compiler, evaluating output, making a hypothesis, running more random strings through the compiler, checking if that proves or disproves the hypothesis, and adjusting the next iteration to generate slightly less random character strings - rinse, repeat. Going through all that effort is stupid, because you could just pick up a book instead - programming is a man-made job, and all the rules are designed in.
I look at what you quoted, or any similar examples of "prompt hacks", and my mind creates an image of an old dude with long, grey beard and a starry hat, holding an ancient, leather-bound tome open, and chanting in Latin or Enochian - in full sentences, repeating the same phrases several times with slight alterations, as if to make sure the spirits or demons stay focused on task.
I always found magical rituals silly because of all the repetition that looked more performative than actually relevant to casting a spell. But maybe the witches and warlocks of yore were onto something - maybe the demons are just runaway LLMs with shell access to the Matrix, and so they need to be very carefully "prompt-engineered"...
EDIT:
For example, imagine Gandalf chanting this:
Now that's obviously just the text from "system_prompt.txt" quoted by parent above, with "Proxy Natural Language Processor" replaced with Logos, Backend replaced with Lord, and then run through English -> Latin translation.