News for hackers. Stuff that matters.

otterk10

Show HN: Realistic Synthetic Conversations for Testing LLMs

https://github.com/Channel-Labs/synthetic-conversation-generation

Testing multi-turn conversational AI is tough, especially when you lack large volumes of real user data. Existing synthetic data tools often generate conversations that lack diversity and are not statistically representative, leading models to overfit synthetic patterns.To help with this problem, I'm open-sourcing a synthetic conversation generation library. This library generates more...

( Read More... | 1 comment | on HN )

Show HN: Product Analytics for LLMs

https://github.com/Channel-Labs/conversational-product-analytics

Over the past two years, I’ve developed a toolkit for helping dozens of clients improve their LLM-powered products, which I'm now open-sourcing. First up: a library to bring product analytics to conversational AI.One of the biggest challenges I see clients face is understanding how their assistants are performing in production. Evals are great for catching regressions, but they can’t...

( Read More... | no comments | 2 points | on HN )

Show HN: Open-Source Conversational Analytics

https://github.com/Channel-Labs/conversational-product-analytics

Over the past two years, I’ve developed a toolkit for helping dozens of clients improve their LLM-powered products, which I'm now open-sourcing.First up: a library to bring product analytics to conversational AI.One of the biggest challenges I see clients face is understanding how their assistants are performing in production. Evals are great for catching regressions, but they can’t surface...

( Read More... | no comments | on HN )
Re:LLM best practices are broken on Thursday February 13, 2025 @12:27PM (#43038586)
I've spent the past 2 years as a fractional AI engineer, and this totally resonates with the advice I've given clients.

They usually hire me after they've gained some initial traction and are stuck on how to improve their LLM further.

Most of the time, they don't have any evals in place. But when they do, it's usually an LLMJudge that shows "everything working well" against a tiny curated dataset. I then spend a couple of weeks painstakingly going through their production data and finding a ton of issues they didn't know about. Their response is usually "holy crap there's so many issues we didn't know about! We had deluded ourselves into thinking our LLM is close to perfect, when it really has a ton of issues".

OpenAI is offering free fine-tuning until September 23rd! To help people get started, I've created an end-to-end example showing how to fine-tune GPT-4o mini to boost the accuracy of classifying customer support tickets from 69% to 94%. Would love any feedback, and happy to chat with anyone interested in exploring fine-tuning further!
Re:Causal Analytics on Friday August 16, 2019 @02:16AM (#20712626)
Thanks for the feedback! I totally agree about observational studies being suggestive but not replacing A/B tests - that’s why the main use case I listed in the blog (and how current customers have used the product so far) is “prioritization of a/b tests”, not replacing a/b tests themselves. The language around “simulating a/b tests” is just a way to try to concisely explain to someone at a high level who may not be very technical or has never heard of an observational study. Happy for suggestions on how to better explain observational studies to less technical customers without over-selling! It’s something we’ve been iterating on ourselves.
Re:Causal Analytics on Friday August 16, 2019 @02:06AM (#20712557)
Thanks for the feedback! I totally agree about observational studies being suggestive but not replacing A/B tests - that’s why the main use case I listed in the blog (and how current customers have used the product so far) is “prioritization of a/b tests”, not replacing a/b tests themselves. The language around “simulating a/b tests” is just a way to try to concisely explain to someone at a high level who may not be very technical or have much experience with causal inference. Happy for suggestions on how to better convey this without over-selling!
Re:Causal Analytics on Friday August 16, 2019 @01:33AM (#20712435)
Thanks! Yes, the concerns you mentioned would also apply to PCA. What we've actually done to help alleviate this is a union of components from y-aware[1] and normal PCA to capture variables that are correlated to both the dependent variables and (hopefully) most of the treatment variables. This seems similar to the double selection approach you mention - the difference being that since we are trying to run this at scale for 1000s of treatment variables, running a feature selection with each of the 1000 treatment variables as the dependent variable isn't super feasible, so the normal PCA acts as proxy for this part of the double selection.

Regardless, we're never going to completely remove omitted variable bias, as we're never going to capture 100% of relevant variables. One way we monitor our model's bias is by looking at the error distribution between users in the treatment vs control. If these aren't similar, there's too much bias in our estimate of the treatment effect, so we wouldn't want to serve an estimate of the treatment effect for this variable to our customers.

The current product is in beta and we're working with some of our current customers to try to re-create our results with A/B tests. I'm hoping that by our GA release in the fall we'll have some case studies with specific examples!

[1] http://www.win-vector.com/blog/2016/05/pcr_part2_yaware/

Re:Causal Analytics on Thursday August 15, 2019 @11:48PM (#20712019)
We didn’t explore causal graphs because doing so would require manually creating a causal graph for each relationship that you wish to explore. Our goal was to create an automated approach that could provide an estimate of the treatment effect for any page/event within your app.
Re:Causal Analytics on Thursday August 15, 2019 @11:24PM (#20711916)
Yes, you're correct that the underlying algorithm used is very close to OLS. What allows the regression to provide an estimate for average treatment effects is how it is structured. Namely, adding in pre-treatment confounders as well as interactions between the treatment and confounders. I found this chapter (http://www.stat.columbia.edu/~gelman/arm/chap9.pdf) on causal inference does a good job of outlining the approach.

Yes, we actually explored other approaches such as PSM. The main reason we did not initially go with PSM was because of the compute power required - you would need to train a model for each treatment variable. However, we're actually in the midst of developing a way to train a model for each treatment variable efficiently, which will allow us to add items such as inverse propensity weighting (or explore other approaches such as PSM).

Re:Causal Analytics on Thursday August 15, 2019 @11:03PM (#20711801)
You can definitely do an AB test to verify the causal relationship - in fact, that's the preferred method! Our platform is for situations where you didn't run an A/B test - either because you can't run as many as you'd like or you forgot - in order to give you an estimate after the fact.
Re:Causal Analytics on Thursday August 15, 2019 @10:50PM (#20711729)
Our analysis runs over our user’s customer data (usually collected through either a tag manager or a CDP such as Segment), which is a few petabytes of data for some of our larger customers. The reason for using Spark is to quickly transform this massive amount of raw data into a ML-ready format. You’re correct that the regression itself does not need to be done inside of Spark.
Re:Causal Analytics on Thursday August 15, 2019 @12:52AM (#20702677)
Thanks! You're correct in surmising that our approach was heavily influenced by Judea Pearl's research.

And yes, the timing of the blog post isn't a coincidence, we actually filed a patent last week :)

Re:Causal Analytics on Wednesday August 14, 2019 @10:23PM (#20702073)
Good to hear! In my experience, BigQuery ML (and other cloud ml products) is great for creating basic models out of the box, but don't provide a ton of flexibility for non-standard ML use-cases. For example, our approach to causal analytics requires doing things such as dimensionality reduction and computing a covariance matrix that are not available through BigQuery ML.

So what we've done instead is create a SparkML task that can read in a feature matrix and trains and scores the causal analytics model. The causal lift estimates for each user are then written out to BigQuery so that in our frontend a customer can filter for, say, users between the ages of 18-35, and then within seconds we'll return them the causal lift of viewing page X for this segment.

Re:Causal Analytics on Wednesday August 14, 2019 @07:25PM (#20701054)
Scott here from ClearBrain - the ML engineer who built the underlying model behind our causal analytics platform.

We’re really excited to release this feature after months of R&D. Many of our customers want to understand the causal impact of their products, but are unable to iterate quickly enough running A/B tests. Rather than taking the easy path and serving correlation based insights, we took the harder approach of automating causal inference through what's known as an observational study, which can simulate A/B experiments on historical data and eliminate spurious effects. This involved a mix of linear regression, PCA, and large-scale custom Spark infra. Happy to share more about what we did behind the scenes!

It’s fascinating to read about the power that Facebook’s advertising apis provide. I’ve always wondered whether Facebook ads are just based off of facebook’s internal data or whether businesses can combine facebook’s data with their own internal data on their customers - it seems like the latter would provide better targeting.