Also would love to hear an answer to this

See reply above^ :)

Don't forget https://benchllm.com/

appreciate it - part of why i put this list up is so that people can add to it lol

Plenty more:

- https://www.getscorecard.ai

- https://arxiv.org/abs/2308.03688

- https://withmartian.com

- https://aihero.studio/

(author of agentops here)

added to my notes! https://github.com/swyxio/ai-notes/

Thanks. I've reviewed all of those solutions and happy to go into more detail in a later post.

Here is our take: None of these solutions are built to fit seamlessly with the data team's workflow to collaborate and work together on (imagine the difference of Modal and WandB had on data / ML teams). We have built and consulted on a number of LLM projects and found that prototypes were super easy to make. Iteration was incredibly difficult. Our goal was to make iteration as easy as writing a unit test.

A number of these projects are missing most Python developer's workflow. For example - we made it compatible with Pytest on day 1 because that's the most common testing suite.

Many competitor tools all make a good pass at these but our roadmap is significantly different and not catered for normal web developers etc. Our focus on building this is to ensure the first step makes sense - that we are super intuitive to add to a data team's Python workflow. (Just 1 assert statement to write a unit test).

cool. thanks for answering!

I'd add ours too, although we're trying to be an end-to-end one-stop platform.

https://github.com/agenta-ai/agenta

you and everyone else, unfortunately. gonna be brutal out there and take exceptional entrepreneurship to be the actual end to end platform that wins

This is a fair question on the surface, but one thing that occurs to me — given the sheer number of similar-looking things people put out, people may not have the time to exhaustively compare their offering with others. You often only discover differentiation by diving deep into each framework/lib etc, and if you dive into every Twitter-amplified new framework/package then you would never have time build your own thing.

So I guess what is a builder to do? Probably focus on practical use cases, show adoption and let these speak for themselves?

for what its worth, i agree, i've had this exact chat with @areibman (who is replying downthread). this is why i asked for market insight rather than a competitor by competitor diff.

Got it, that makes sense

Are there any large or publically shared evaluation datasets and corresponding question and answer validation data?

A few publically shared evaluation datasets include:

- HuggingFace Evaluate (and their 3 dashboards) - OpenAIEval - AnthropicEval

I would not use those for production.

I would assume your backend model is already tested on these.

You would need a novel data set.