News for hackers. Stuff that matters.

Ask HN: How does one build an LLM router with a labelled set of 100 tasks?

I already spent the time building rubrics for a set of ~100 tasks on various criterias (accuracy, thoroughness communication style ). Then ran these on a few models and now have a labelled-set where I know for each task which is the cheapest/fastest model that gets a minimum threshold ( say 80 ).

How does one go on to build a router based on this ? There appears to be no good literature / open source projects on this.

I tried a bunch of approaches with the help of claude: - trying to use Jev via a serious of probe Yes/No questions ( does task contain x, does the world have y ...etc ). But this didn't yield good results. - kNN on embeddings of the task - in retrospect this was set to fail from the tasks, as now I think we do need some sort of NN to figure out the world/task complexity. - classifying prompt-complexity - something like https://catalog.ngc.nvidia.com/orgs/nvidia/nemo/models/prompt-task-and-complexity-classifier/-

The main issue from a conceptual standpoint is that one cannot fully predict how hard a task is until you are into the task - like you might encounter unknown nuances or issues in later stages, but classification has to happen at start. So I am starting to doubt if this is even possible tbh

( no comments | on HN )

There are no comments yet.