The Problem with LangChain

Re:The Problem with LangChain by simonw on Friday July 14, 2023 @01:44PM (#36727388)

I added a Python library API to my LLM CLI tool recently which offers a very lightweight way to call models: https://llm.datasette.io/en/stable/python-api.html

    import llm
    model = llm.get_model("gpt-3.5-turbo")
    model.key = 'YOUR_API_KEY_HERE'
    response = model.prompt(
        "Five surprising names for a pet pelican"
    )
    print(response.text())

Or you can stream the responses like this:

    response = model.prompt(
        "Five diabolical names for a pet goat"
    )
    for chunk in response:
        print(chunk, end="")

It works with other models too, installed via plugins - including models that can run directly on your machine:

    pip install llm-gpt4all

Then:

    model = llm.get_model("ggml-vicuna-7b-1")
    print(model.prompt(
        "What is the capital of France?"
    ).text())

I wrote more about the new plugin system for adding extra models here: https://simonwillison.net/2023/Jul/12/llm/

[ Parent | on HN ]

Re:The Problem with LangChain by sitkack on Friday July 14, 2023 @07:25PM (#36731325)

Why is the conversation tied to the model?

[ Parent | on HN ]

Re:The Problem with LangChain by simonw on Friday July 14, 2023 @08:16PM (#36731756)

Good question! It's because there's an aspect of conversations that differs between different models: the way the previous messages are injected into the context of the prompt.

I went back and forth on a bunch of different designs, but eventually decided to try to make it so that each plugin that implemented a new model would only have to subclass Model and add new methods.

You can see all of the design arguments I had with myself about this here, across the course of 129 pull requests comments: https://github.com/simonw/llm/pull/65

[ Parent | on HN ]

Re:The Problem with LangChain by brianjking on Friday July 14, 2023 @08:47PM (#36731971)

Super pumped about this, Simon. Will be trying it this weekend. Any of the gpt4all models work, but need to duplicate the model files for now, right?

[ Parent | on HN ]

Re:The Problem with LangChain by simonw on Friday July 14, 2023 @09:15PM (#36732129)

Yeah I haven't figured out how to have it reuse the models from the desktop GPT4All installation yet, issue here: https://github.com/simonw/llm-gpt4all/issues/5

[ Parent | on HN ]

Re:The Problem with LangChain by bspammer on Saturday July 15, 2023 @04:01AM (#36734608)

This looks wonderful, a similar breath of fresh air to using the requests library for the first time. Really impressed by the amount of documentation too.

Is support for embedding and querying a corpus of custom text planned at all? 99% of what I wanted to use langchain for was building a chatbot that can answer questions about my own documents.

[ Parent | on HN ]

Re:The Problem with LangChain by simonw on Saturday July 15, 2023 @10:08AM (#36736898)

Undecided yet, but I think there's a good chance embedding stuff will eventually show up as an LLM plugin.

[ Parent | on HN ]

Re:The Problem with LangChain by rcarmo on Saturday July 15, 2023 @06:37AM (#36735357)

Nice. Now all we need is a vector database atop SQLite.

[ Parent | on HN ]

Re:The Problem with LangChain by bread90 on Saturday July 15, 2023 @06:46AM (#36735405)

Just implemented pgvector in an existing psql db, works quite well

[ Parent | on HN ]

Re:The Problem with LangChain by simonw on Saturday July 15, 2023 @10:07AM (#36736889)

I had a go at one of those a few months ago: https://datasette.io/plugins/datasette-faiss

Alex Garcia built a better one here as a SQLite Rust extension: https://github.com/asg017/sqlite-vss

[ Parent | on HN ]