News for hackers. Stuff that matters.

varunkrishnan17

Show HN: RL Agent that can auto-optimize your LLM prompts

https://nomadic-ml.github.io/nomadic/cookbooks/Nomadic_Prompt_Optimization_Report.html

Hey everyone! Along with my team, I've developed a reinforcement learning system that automatically optimizes LLM prompts, complete with a visualization feature to track both prompt structure and learning progress over time. Take a look here: https://nomadic-ml.github.io/nomadic/cookbooks/Nomadic_Promp...Check out our website too:https://www.nomadicml.com/In terms of how this visualization...

( Read More... | 2 comments | 8 points | on HN )
Really appreciate it Jadiker! We're obviously in a similar space, but we think we offer some strong differentiators: (1) Functionality that is more specific to your LLM use cases (for example, being able to easily kick off a RAG Retrieval / Inferencing Experiment). (2) Ability to easily customize and visualize your results - for example, through custom evaluators, and carefully curated heatmaps - both through our SDK and our managed service.
Thanks for the well-thought out question Jadiker!

This is a potential limitation of N-gram precision with context matching, which we were using in the RAG demo for simplicity (though even with this, I don't think it would be so extreme :-) )

We already offer two other different hallucination detection approaches which should mitigate this problem - an LLM-as-a-judge model for evaluation, and semantic similarity matching. We've also considered, for example, using metrics such as BertScore. Do you have other ideas? :-)

That's a great use case of Nomadic! We support many Eval frameworks in the optimization, but one is a LLM-as-a-Judge model, where you can input custom weights based on your metrics of interest! Adhering to a proper structure could be one of them :-)
Hi I'm Varun - one of the cofounders of Nomadic!

Been a pleasure to work with Mustafa and Lizzie on this! Hopefully you can solve a pain point I personally have had for so long - how can you easily verify that your model continues to perform well?