Hey, cofounder here. As far as I know, LangSmith focuses on observability (token usage, chain of thought insights, LLM cost), is vendor locked into LangChain, and uses LLMs to evaluate themselves.

Here are the problems with what LangSmith offers: 1. Focusing on observability brings insights to engineering teams in terms of cost, latency, but these conclusions aren't really actionable

2. Not all teams use LangChain, plus a lot of teams who originally used LangChain to prototype is moving away as they productionize and look to improve what they've built

3. Not every LLM implementation involves a chain of thought, lLamaIndex is a good example. Apart from vendor locking, there are other use cases such as evaluating fine tuned models that are overlooked in a closed source software like LangSmith

4. Using LLMs to evaluate themselves are fine, but a lot more can be done. We offer other metrics (metrics that can be quantified from 0 - 1) such as factual consistency trained on NLI models that are much more deterministic. You can also use our package to pick and choose metrics that you care about (bias, relevancy, toxicity, etc)

Hope that answers your questions.