News for hackers. Stuff that matters.

okhat

This is a DSPy optimizer, built by the DSPy core team. Just wait for open sourcing.
I'll admit even I sometimes wish ColBERT was more user-friendly. I'll probably start using ColBERT throught RAGatouille now.
Writing right now. This month :-)
Thanks! Lots to discuss from your excellent response, but I'll address the easy part first: DSPy is v2 of DSP (demonstrate-search-predict).

The DSPy paper hasn't been released yet. DSPy is a completely different thing from DSP. It's a superset. (We actually implemented DSPy _using_ DSPv1. Talk about bootstrapping!)

Reading the DSPv1 paper is still useful to understand the history of these ideas, but it's not a complete picture. DSPy is meant to be much cleaner and more automatic.

@wokwokwok Okay now we disagree. This task is not easy, it's just easy to follow in one notebook. (If it were easy, the RAG score wouldn't be 26%.)

As for "carefully crafted string templates", I'm not sure what your argument here is. Are you saying you could have spent a few hours of trial and error writing 3 long prompts in a pipeline, until you matched what the machine does in 60 seconds?

Yes, you probably could have :-)

Ah okay makes sense, yeah we'll release more examples.

This is just an intro to the key concepts/modules.

What did you find underwhelming if I may ask?

It shows you how it takes some ~25 Pythonic lines of code to make GPT-3.5 retrieval accuracy go from the 26-36% range to 60%.

Not a bad deal when you apply it to your own problem?

Here's the key idea.

You give DSPy (1) your free-form code with declarative calls to LMs, (2) a few inputs [labels optional], and (3) some validation metric [e.g., sanity checks].

It simulates your code on the inputs. When there's an LM call, it will make one or more simple zero-shot calls that respect your declarative signature. Think of this like a more general form of "function calling" if you will. It's just trying out things to see what passes your validation logic, but it's a highly-constrained search process.

The constraints enforced by the signature (per LM call) and the validation metric allow the compiler [with some metaprogramming tricks] to gather "good" and "bad" examples of execution for every step in which your code calls an LM. Even if you have no labels for it, because you're just exploring different pipelines. (Who has time to label each step?)

For now, we throw away the bad examples. The good examples become potential demonstrations. The compiler can now do an optimization process to find the best combination of these automatically bootstrapped demonstrations in the prompts. Maybe the best on average, maybe (in principle) the predicted best for a specific input. There's no magic here, it's just optimizing your metric.

The same bootstrapping logic lends itself (with more internal metaprogramming tricks, which you don't need to worry about) to finetuning models for your LM calls, instead of prompting.

In practice, this works really well because even tiny LMs can do powerful things when they see a few well-selected examples.

These are answers to specific questions below.
``` ### DSPy vs. application development libraries like LangChain, LlamaIndex

LangChain and LlamaIndex are popular libraries that target high-level application development with LMs. They offer many batteries-included, pre-built application modules that plug in with your data or configuration. In practice, indeed, many usecases genuinely don't need any special components. If you'd be happy to use someone's generic, off-the-shelf prompt for question answering over PDFs or standard text-to-SQL as long as it's easy to set up on your data, then you will probably find a very rich ecosystem in these libraries.

Unlike these libraries, DSPy doesn't internally contain hand-crafted prompts that target specific applications you can build. Instead, DSPy introduces a very small set of much more powerful and general-purpose modules that can learn to prompt (or finetune) your LM within your pipeline on your data.

DSPy offers a whole different degree of modularity: when you change your data, make tweaks to your program's control flow, or change your target LM, the DSPy compiler can map your program into a new set of prompts (or finetunes) that are optimized specifically for this pipeline. Because of this, you may find that DSPy obtains the highest quality for your task, with the least effort, provided you're willing to implement (or extend) your own short program. In short, DSPy is for when you need a lightweight but automatically-optimizing programming model — not a library of predefined prompts and integrations.

If you're familiar with neural networks:

This is like the difference between PyTorch (i.e., representing DSPy) and HuggingFace Transformers (i.e., representing the higher-level libraries). If you simply want to use off-the-shelf BERT-base-uncased or GPT2-large or apply minimal finetuning to them, HF Transformers makes it very straightforward. If, however, you're looking to build your own architecture (or extend an existing one significantly), you have to quickly drop down into something much more modular like PyTorch. Luckily, HF Transformers is implemented in backends like PyTorch. We are similarly excited about high-level wrapper around DSPy for common applications. If this is implemented using DSPy, your high-level application can also adapt significantly to your data in a way that static prompt chains won't.

```

just posted a top-level answer, copied from the FAQs of DSPy
"print out every prompt that is generated to a log" --- yes of course

This Colab is full of prompts and examples of improving the quality of gpt-3.5-turbo: https://t.co/Oa1RDp3XbZ

Paper incoming, but basically we've seen > 50% quality gains by just compiling in various settings.

This Twitter/X thread discusses doing a simple program for Llama2, with massive quality gains too: https://twitter.com/lateinteraction/status/16947484013744909...

@simonw it sounds like we'd agree that:

1] when prototyping, it's useful to not have to tweak each prompt by hand as long as you can inspect them easily

2] when the system design is "final", it's important to be able to tweak any prompts or finetunes with full flexibility

But we may or may not agree on:

3] automatic optimization can basically make #2 above only very rarely needed

---

Anyway, the entire DSPy project has zero hard-coded prompts for tasks. It's all bootstrapped and validated for your logic. In case you're worried that we're doing some opinionated prompting on your behalf.

``` ### DSPy vs. thin wrappers around prompts (OpenAI API, MiniChain, basic templating, etc.)

In other words: Why can't I just write my prompts directly as string templates? Well, for extremely simple settings, this might work just fine. (If you're familiar with neural networks, this is like expressing a tiny two-layer NN as a Python for-loop. It kinda works.)

However, when you need higher quality (or manageable cost), then you need to iteratively explore multi-stage decomposition, improved prompting, data bootstrapping, careful finetuning, retrieval augmentation, and/or using smaller (or cheaper, or local) models. The true expressive power of building with foundation models lies in the interactions between these pieces. But every time you change one piece, you likely break (or weaken) multiple other components.

DSPy cleanly abstracts away (and powerfully optimizes) the parts of these interactions that are external to your actual system design. It lets you focus on designing the module-level interactions: the same program expressed in 10 or 20 lines of DSPy can easily be compiled into multi-stage instructions for GPT-4, detailed prompts for Llama2-13b, or finetunes for T5-base.

Oh, and you wouldn't need to maintain long, brittle, model-specific strings at the core of your project anymore. ```

"A neural network layer is just a matrix. Why abstract that matrix and learn it?" Well, because it's not your job to figure out how to hardcode delicate string or floats that work well for a given architecture & backend.

We want developers to iterate quickly on system designs: How should we break down the task? Where do we call LMs? What should they do?

---

If you can guess the right prompts right away for each LLM, tweak them well for any complex pipeline, and rarely have to change the pipeline (and hence all prompts in it), then you probably won't need this.

That said, it turns out that (a) prompts that work well are very specific to particular LMs, large & especially small ones, (b) prompts that work well change significantly when you tweak your pipeline or your data, and (c) prompts that work well may be long and time-consuming to find.

Oh, and often the prompt that works well changes for different inputs. Thinking in terms of strings is a glaring anti-pattern.

DSPy provides composable and declarative modules for instructing LMs in a familiar Pythonic syntax and an automatic compiler that teaches LMs how to conduct the declarative steps in your program. Specifically, the DSPy compiler will internally trace your program and then craft high-quality prompts for large LMs (or train automatic finetunes for small LMs) to teach them the steps of your task.
Re:Re-implementing LangChain in 100 lines of code on Thursday May 04, 2023 @09:59PM (#35824459)
There’s always DSP for those who need a lightweight but powerful programming model — not a library of predefined prompts and integrations.

It’s a very different experience from the hand-holding of LangChain, but it packs reusable magic in generic constructs like annotate, compile, etc that work with arbitrary programs.

https://github.com/stanfordnlp/dsp/

Re:Prompt Engine – Microsoft's prompt engineering library on Thursday February 16, 2023 @05:35AM (#34817066)
Very cool! But for hard enough problems, prompt engineering is kind of like hyperparameter tuning. It's only a final (and relatively minor) step after building up an effective architecture and getting its modules to work together.

DSP provides a high-level abstraction for building these architectures—with LMs and search. And it gets the modules working together on your behalf (e.g., it annotates few-shot demonstrations for LM calls automatically).

Once you're happy with things, it can compile your DSP program into a tiny LM that's a lot cheaper to work with.

https://github.com/stanfordnlp/dsp/