matt_lee

When we got accepted into S23, our call started with Seibel saying something along the lines of "We like you, but we're pretty sure you're going to have to pivot." A non-trivial amount of my batch mates also joined with minimal product strategy, so I don't think you can read too much into the rejections.

I don't have anything to add re: YC over the years, just that my anecdotal experience differs.

> Why not "why not" comments? Not why "not comments"

I nearly exploded trying to grok this

A few weeks back we shared a demo [1] of this on HN and got to chat with a lot of interesting folks but we weren't able to support everyone just because we're a small team.

We're opening up a free developer tier of our CLI tool in hopes that it can be helpful to more people! We gated it behind email verification just to CYA because we're using our own LLM API calls and compute behind the scenes.

If you have any issues please comment here or shoot me a note at matt at talc ai.

[1] https://news.ycombinator.com/item?id=39042093

EDIT: I'm also just noticing that in the blog text I wrote "generated for Boeing". I meant Boeing as a topic, not as a company-- sorry for the sloppy language. I'll push a fix for this later today.

Thank you for the kind words!

One of my regrets about the demo is that we paid a lot of attention to showing off our ability to generate high quality Q/A pairs, but not nearly as much to showing what a thoughtful and thorough grading rubric can do.

It's totally possible to do a high quality grading given a rubric that sets expectations! Great implementations we've seen use categories like correct / correct but incomplete / correct but unhelpful / incorrect to better label the situation you describe. We've found that we can grade with much more nuance given a good rubric and categories, but unfortunately didn't focus on that side of things in the demo

I'm not familiar with wikicrow, will check it out!

Not a pricing expert by any means, but here were some of the considerations we thought of when we made that decision:

1. There's plenty of usage based infrastructure/dev tools (ie. AWS, databricks), so I don't think we incentivize minimal usage.

2. The value we're providing feels directly tied to how much testing we're running, so when we tried to construct tiers they didn't feel helpful.

3. In our experience, enterprises have been fine with usage based pricing. They're already paying for human QA / labelling on usage based terms (even if it's part of a larger fixed contract), so our pricing isn't a deviation for them.

Open to thoughts if anyone has them!

Thanks for giving it a whirl!

I agree that the current grading is a bit harsh -- the rubric we're using in this demo is fairly rudimentary. What we've seen be more helpful is a range of grades along the lines of correct / correct but unhelpful / correct but incomplete / incorrect. This somewhat depends on individual use cases though.

Let me know what questions generated you thought could be more complex! We're always working on improving our ability to explore the knowledge space for challenging questions.

Thanks! Max's cofounder chiming in here.

1) There's an interesting subtlety in the phrase "the correct question-answer pairs". While we don't often find factually incorrect pairs because of how we're running the pipeline, the bigger question is whether or not the pairs we generated are "the" correct ones -- if they are relevant and helpful. This takes some manual tweaking at the moment.

Inconsistent outputs over different runs are definitely an issue, but most teams we've worked with barely even have the CI/CD practice to be able to measure that rigorously. As we mature we'll aim to tackle flakiness of tests (and models) over time, but a bigger challenge has been getting regular tests like these set up in the first place.

2) In this scenario, we go to the documents powering a RAG application to both generate and grade answers. For example, the knowledge base might know that (1) product A is being recalled, and (2) customer #4 is asking for a warranty claim on product A. Using those two bits of information, we might generate a scenario that tests whether or not customer #4 gets the claim fulfilled. In other words, specific user information is simulated/used during the test set creation.