jneagu

Hey @echollama! Can you share more about what kind of integrations you are looking for? Also, when you say "enable customers to configure their usage" - what types of configurations would you want? How are you doing all of this today?

"trained on licensed content" - this is a bit misleading. The correct framing is in the article: "made with content owner permission." Most (all?) LLMs out there are trained on licensed content, they just try to catch and filter it before it's outputted to the user.

I can save you a bit of market research and tell you that’s unfortunately not the case yet in the market today. There are a few reasons for it - the main one in my opinion being that it’s hard to measure the value vs cost of switching to an open LLM, so it’s generally perceived as same/lower value, higher cost (not in terms of inference, but in terms of overhead). What is considered however are cost saving options around the foundational models: going for the mini models, prompt caching, batch inference etc. Some tooling in that area might be interesting.

The challenge with A/B experiments is how you design them to have sufficient power and draw a meaningful conclusion out of them. So, you either need a big % difference between the test and the control, or you need a big number of samples. LLM apps usually don’t meet either of those two criteria. Have you ran into this with your users?

Nice! Overall, frameworks that re-prompt an LLM with feedback or failure modes of the original outputs do very well.

To your latter point - that’s where I think most of the value of LLMs in education is. They can explain code beyond the educational content that’s already available out there. They are pretty decent at finding and explaining code errors. Someone who’s ramping up their coding skills can make a lot of progress with those two features alone.

I am very curious to see how this is going to impact STEM education. Such a big part of an engineer's education happens informally by asking peers, teachers, and strangers questions. Different groups are more or less likely to do that consistently (e.g. https://journals.asm.org/doi/10.1128/jmbe.00100-21), and it can impact their progress. I've learned most from publicly asking "dumb" questions.

Anecdotally, synthetic data can get good if the generation involves a nugget of human labels/feedback that gets scaled up w/ a generative process.

Fair point - I actually had parsed OP's sentence differently. I'll edit my comment.

I agree, LLMs performance for coding tasks is super biased in favor of well-represented languages. I think this is what GitHub is trying to solve with custom private models for Copilot, but I expect that to be enterprise only.

Yeah, There was a reference in a paywalled article a year ago (https://www.theinformation.com/articles/openai-made-an-ai-br...): "Sutskever's breakthrough allowed OpenAI to overcome limitations on obtaining high-quality data to train new models, according to the person with knowledge, a major obstacle for developing next-generation models. The research involved using computer-generated, rather than real-world, data like text or images pulled from the internet to train new models."

I suspect most foundational models are now knowingly trained on at least some synthetic data.

Interesting - I actually think they perform quite well on code, considering that code has a set of correct answers (unlike most other tasks we use LLMs for on a daily basis). GitHub Copilot had a 30%+ acceptance rate (https://github.blog/news-insights/research/research-quantify...). How often does one accept the first answer that ChatGPT returns?

To answer your first question: new content is still being created in an LLM-assisted way, and a lot of it can be quite good. The rate of that happening is a lot lower than that of LLM-generated spam - this is the concerning part.

Hey HN,

We are excited to reveal Quotient, a platform enabling developers to evaluate, improve and ship high-quality AI products through fast, real-world, data-backed experimentation. [1]

Every developer understands that for a change in their system to be strategic rather than impulsive, it is crucial and essential to test its impact on the desired outcome. The ability to do this seamlessly, swiftly, and reliably results in flywheels of iterative improvements, which ultimately create disruptive products that delight users.

Quotient equips all AI developers with the tools to achieve just that. With Quotient, you can swiftly gauge the effectiveness of their generative AI solutions to meet the unique requirements of each user or customer.

Quotient offers AI developers the capability to evaluate their AI products with specialized datasets and frameworks, which can be customized or ready-to-use, and tailored to their organization. By addressing challenges in evaluation management, version control, and orchestration, Quotient streamlines assessments of models, prompts, and retrieval augmentation through rapid, data-driven iteration cycles.

Our vision for Quotient is rooted in our experience developing the evaluation infrastructure behind GitHub Copilot. You can read more about our mission and founding story here.

We are currently providing free access to our beta, with enterprise-grade plans available to our power users. You can request access through our website. [2]

We are also hiring! [3]

Finally, if you are working on an AI experiment, project, or product, and are (stuck) thinking about testing and improvements – we want to hear from you! Shoot us a note at [email protected] with some info about your work – we’ll schedule some time to chat and see how we can help.

We’re excited to hear from you, and see what we can achieve, together.

[1] https://quotientai.co/post/hello-world-were-quotient [2] https://www.quotientai.co [3] https://www.linkedin.com/company/quotient-ai/jobs/