hellovai

(on of the creators of BAML here) yep! exactly!

that workaround we've found works quite well, but the problem is that its not sufficient to just retry in the case of failed schema matches (its both inefficient and also imo incorrect).

Take these two scenarios for example:

Scenario 1. My system is designed to output receipts, but the user does something malicious and gives me an invoice. during step 2, it fails to fit the schema, but then you try with step 3, and now you have a receipt! Its close, but your business logic is not expecting that. Often when schema alignment fails, its usually because the schema was ambiguous or the input was not valid.

Scenario 2. I ask the LLM to produce this schema:

    class Person {
      name string
      past_jobs string[]
    }

However the person only has ever worked at 1 job. so the LLM outputs: { "name": "Vaibhav", "past_jobs": "Google" }. Technically since you know you expect an array, you could just transform the string -> string[].

thats the algorithm we created: schema-aligned parsing. More here if you're interested: https://boundaryml.com/blog/schema-aligned-parsing

Benchmark wise, when we tested last, it seems to help on top of every model (especially the smaller ones) https://www.reddit.com/r/LocalLLaMA/comments/1esd9xc/beating...

Hope this helps with some of the ambiguities in the post :)

if you haven't tried the research -> plan -> implementation approach here, you are missing out on how good LLMs are. it completely changed my perspective.

the key part was really just explicitly thinking about different levels of abstraction at different levels of vibecoding. I was doing it before, but not explicitly in discrete steps and that was where i got into messes. The prior approach made check pointing / reverting very difficult.

When i think of everything in phases, i do similar stuff w/ my git commits at "phase" levels, which makes design decision easier to make.

I also do spend ~4-5 hours cleaning up the code at the very very end once everything works. But its still way faster than writing hard features myself.

Have you tried techniques that don’t require modifying the LLM and the sampling strategy for structure outputs? For example, schema aligned passing, where you build error tolerance into the parser instead of coercing to a grammar.

https://boundaryml.com/blog/schema-aligned-parsing

if you share your prompt with me on promptfiddle.com i can play around with it and see how i can make it better!

its a bit more nuanced than applicative lifting. parts of of SAP is that, but there's also supporting strings that don't have quotation marks, supporting recursive types, supporting unescaped quotes like: `"hi i wanted to say "hi""`, supporting markdown blocks inside of things that look like "json", etc.

but applicative lifting is a big part of it as well!

gloochat.notion.site/benefits-of-baml

appreciate you tyring it. the reason it dropped the day was due to your type system not being understood by the LLM you're using.

the model replied with

       {
          "Text": "coffee liqueur",
          "Type": "Liqueur",
          "Liquor_type": "Liqueur",
          "Name_brand": null,
          "Unit_of_measure": "ounce",
          "Measurement_or_unit_count": "3/4"
        },

but you expected a { Text: string, Type: IngredientType, Liquor_type: LiquorType or null, Name_brand: string or null, Unit_of_measure: string, Measurement_or_unit_count: string, }

there's no way to cast `Liqueur` -> `IngredientType`. but since the the data model is a `Ingredient[]` we attempted to give you as many ingredients as possible.

The model itself being wrong isn't something we can do much about. that depends on 2 things (the capabilities of the model, and the prompt you pass in).

If you wanted to capture all of the items with more rigor you could write it in this way:

    class Recipe {
        name string
        ingredients Ingredient[]
        num_ingredients int
        ...

        // add a constraint on the type
        @@assert(counts_match, {{ this.ingredients|length == this.num_ingredients }})
    }

And then if you want to be very wild, put this in your prompt:

   {{ ctx.output_format }}
   No quotes around strings

And it'll do some cool stuff

have you tried schema-aligned parsing yet?

the idea is that instead of using JSON.parse, we create a custom Type.parse for each type you define.

so if you want a:

   class Job { company: string[] }

And the LLM happens to output:

   { "company": "Amazon" }

We can upcast "Amazon" -> ["Amazon"] since you indicated that in your schema.

https://www.boundaryml.com/blog/schema-aligned-parsing

and since its only post processing, the technique will work on every model :)

for example, on BFCL benchmarks, we got SAP + GPT3.5 to beat out GPT4o ( https://www.boundaryml.com/blog/sota-function-calling )

really cool to see BAML on here :) 100% align on so much of what you've said here. its really about treating LLMs as functions.

yea! even deepseek. Calling an external function / tool calling is really just a data extraction problem.

say you have a tool:

def calculator(expr: str) -> float

then the model just needs to say:

{ "function": "calculator", "args": { "expr": "5 + 10" } }

then in your code you can easily pass that to the "calculator" function and get the result, then hand the result back to the model. Making it feel like the model can "call" an external function.

deep seek can also do this: https://www.boundaryml.com/blog/deepseek-r1-function-calling

i 100% percent agree. people get so caught up on trying to do everything 90% right with AI, but they forget there's a reason most websites offer at least 2 9's of uptime.

We have some preliminary data with llama3.1 and we find that the smaller model gets to around 70% with BAML (+20% from base), but we'll update this dashboard with llama3.1 by end of week!

Take a look at BAML (boundaryml.com)

Its a different take that leverages a DSL to make prompting cleaner and fixes a few other ergonomic issues along the way.

you can try it online at promptfiddle.com

nothing specific, but you can try our prompt / datamodel out on https://www.promptfiddle.com

or if you're open to share your prompt / data model with, I can send over my best guess of a good prompt! We've found these models works even with over 50+ fields / nested and whatnot decently well!

we recently added dynamic type support with this snippet! (docs coming soon!)

Python: https://github.com/BoundaryML/baml/blob/413fdf12a0c8c1ebb75c...

Typescript: https://github.com/BoundaryML/baml/blob/413fdf12a0c8c1ebb75c...

Snippet:

async def test_dynamic():

    tb = TypeBuilder()

    tb.Person.add_property("last_name", tb.string().list())

    tb.Person.add_property("height", tb.float().optional()).description(
        "Height in meters"
    )


    tb.Hobby.add_value("chess")

    for name, val in tb.Hobby.list_values():
        val.alias(name.lower())

    tb.Person.add_property("hobbies", tb.Hobby.type().list()).description(
        "Some suggested hobbies they might be good at"
    )

    # no_tb_res = await b.ExtractPeople("My name is Harrison. My hair is black and I'm 6 feet tall.")
    tb_res = await b.ExtractPeople(
        "My name is Harrison. My hair is black and I'm 6 feet tall. I'm pretty good around the hoop.",
        {"tb": tb},
    )

    assert len(tb_res) > 0, "Expected non-empty result but got empty."

    for r in tb_res:
        print(r.model_dump())

oh thats really interesting, how often do you get errors like that?

fyi, we actually fix those specific errors in our parser :)

The main drawback is really when you attempt to do more advanced prompting techniques like chain-of-thought or reasoning.

forcing those parts to be json, can be hard and unnecessarily constrain the model. e.g. https://www.promptfiddle.com/Chain-of-Thought-KcSBh

try pressing run tests and you'll see what i mean! this method or doing chain of thought works a bit better

;) https://www.promptfiddle.com/structured-summary-66myE (sorry bad syntax highlighting when including baml code in baml code)

{ author: "Sam Lijin"

key_points: [ "Structured output from LLMs, like JSON, is a common challenge."

  "Existing solutions like response_format: 'json' and function calling often disappoint."

  "The article compares multiple frameworks designed to handle structured output."

  "Handling and preventing malformed JSON is a critical concern."

  "Two main techniques for this: parsing malformed JSON or constraining LLM token generation."

  "Framework comparison includes details on language support, JSON handling, prompt building, control, model providers, API flavors, type definitions, and test frameworks."

  "BAML is noted for its robust handling of malformed JSON using a new Rust-based parser."

  "Instructor supports multiple LLM providers but has limitations on prompt control."

  "Guidance, Outlines, and others apply LLM token constraints but have limitations with models like OpenAI's."

]

take_way: "Consider using frameworks that efficiently handle malformed JSON and offer prompt control to get the desired structured output from LLMs."

}

that's a great question, there's three main benefits:

1. seeing the full prompt, even though that python code feels leaner, somehow you need to convert it to a prompt. a library will do that in some way, BAML has a VSCode playground to see the entire prompt + tokenization. If we had to do this off of python/ts, we would run into the halting problem and making the playground would be much much harder.

2. there's a lot of codegen we do for users, to make life easier, e.g. w/o BAML, to now do streaming for the resume, you would have to do something like this:

class PartialResume: name: Optional[str] education: List[PartialEducation] skills: List[str]

and then at some point you need to reparse PartialResume -> Resume, we can codegen all of that for you, and give you autocomplete, type-safety for free.

3. We added a lot of static analysis / jump to definition etc to JINJA (which we use for strings), and that is much easier to navigate than f-strings.

4. Since its code-gen we can support all languages way easier, so prompting techniques in python work the same exact way for the same code in typescript.

the main one is that most people don't own the model. so if you use openai / anthropic / etc then you can't use token masking. in that case, reprompting is pretty much the only option

thats pretty cool! We'll update the page after taking a look at the library!

our paid product is still in Beta actually as we're continuing to build it out, but BAML itself is and always will be open source (runs fully locally as well - no extra network calls).

in terms of parsing, I do think we're likely the best approach as of now. Most other libraries do reprompting or rely on constraining grammars which require owning the model. Reprompting = slow + $$, constraining grammars = require owning the model. we just tried a new approach: parse the output in a more clever way.

Thanks! We should add that to the docs haha. But the here's a few:

- keys without strings

- coercing singular types -> arrays when the response requires an array

- removing any prefix or suffix tags

- picking the best of many JSON candidates in a string

- unescaped newlines + quotes so "afds"asdf" converts to "afds\"asdf"

In terms of models, honestly, we tried as bad as llama2, and it seems to work in quite a few use cases

not a noob question, here's how the LLM works:

```

prompt = "..."

output = []

do:

  token_probabilities = call_model(prompt)

  best_token = pick_best(token_probabilities)

  if best_token == '<END>':

    break

  output += best_token

while true

return output

```

basically to support generation they would need to modify pick_best to support constraining. That would make it so they can't optimize the hot loop at their scales. They support super broad output constraints like JSON which apply to everyone, but that leads to other issues (things like chain-of-thought/reasoning perform way worse in structured responses).

XML is also a great option, but there are a few trade offs:

> XML is a many more tokens (much slower + $$$ for complex schemas)

> regardless of if you're looking for } or </output> its really a matter of "does your parser work". when you have three tokens that need to be correct "</" "output", ">", the odds of a mistake are higher, instead of when you just need "}".

That said, the parser is much easier to write, we're actually considering supporting XML in BAML. have you found any reductions of accuracy?

Also, not sure if you saw this, but apparently Claude doesn't actually prefer XML, it just happens to work well with it. Was recently new info for myself as well. https://x.com/alexalbert__/status/1778550859598807178 (devrel @ Anthropic)