mfkhalil

we still browse issues to check for anything critical but this is really useful for uncovering patterns / symptoms of the same root cause.

also it gets issue body + comment/reaction metadata so it can generally intuit whether something is important or not

It’s a mix. On benchmarks, yeah, we route a request, then run the same request on models within a range of capability and compare the results.

We also have our entire team using it for their day-to-day work, and they’re devs who are pumping out tens of PRs a day, so they’re a pretty unforgiving test group and are always giving feedback.

We also work with companies that have it either running on real traffic or running shadow evals in the background (which do a version of the +/-1 method you’re describing), and they’re a huge source of feedback.

At the end of the day, most of our focus is on how it performs on production workloads. As useful as benchmarks can be for initial testing, I think it’s dangerous to over-optimize for those kinds of tasks (and most frontier models have already learned how to solve those problems optimally during training, so results can be misleading there).

Hey, I'm on the team at LiteLLM that's building the auto-router and our goal right now is to abstract that decision making away from the end user. The biggest thing we're trying to figure out right now is how do we do that without frustrating the end user - as a developer myself I would hate for my agent to be dumbed down below the threshold needed to complete a task.

In theory though, there is a minimum viable model for any given task, and we think that is a problem that the big labs will avoid because they profit from charging more per task. We're trying heuristic and LLM-based approaches but it's still a work in progress, so if this is something you'd be interested in trying would highly recommend trying ours out -- any and all feedback at this point is extremely valuable to us.

https://docs.litellm.ai/docs/proxy/auto_routing

Very salient point. My take on this (and I know it’s not very popular on HN) is that trying to preserve every pre-AI coding skill is probably a fruitless endeavor.

AI-assisted coding is here and it’s not going anywhere. In the same way most modern software engineers aren’t writing assembly anymore, I don’t think future engineers will need to understand exactly how every part of a codebase works under the hood. It’s just another layer of abstraction.

Having said that, I do think knowledge of architecture/performance/security remains a pretty large part of prompting correctly, although more conceptually than actual implementation. As long as you’re still using those concepts in your prompting, you can keep that judgment sharp, even if some of the implementation fluency fades.

For example, with the boxing analogy, yes, you lose some boxing skills, but you’re also becoming a way better boxing coach by consistently teaching it. I think that’s the tradeoff.

Built this internally for our coding agents but have been loving it so much that we decided to make it public.

The web UI is free to use and does not require sign up, and has every public SKILL.md on GitHub indexed, with the index refreshing every 48 hours.

Programmatic use is $20 a month for unlimited searches, and I can vouch for the fact that it's made my agents feel so much smarter and more knowledgeable. Only issue is right now you have to nudge it to use skillhound ("use skillhound first") otherwise it tends to try to figure out best practices on its own.

Hope it can be as useful to the community as it is to us, and would appreciate any feedback you have.

We're working on Webhound - budget controlled long-running deep research. You set a budget and Webhound will use that much in compute/LLM tokens to research your prompt, with built in verification cycles and optional added verification budget. Every claim is cited with evidence and a direct link to the tool calls that produced the claim

The goal is to build a deep research product for actual researchers, since we believe that it is an extremely powerful product that is still nascent but has enormous potential - which we've already seen with some early users.

https://webhound.ai

The least productive teams I've been a part of are the ones where everyone is waiting for their turn to say why an idea is bad. Sometimes being "too smart" can hold you back from building something genuinely new.

Hey, appreciate the feedback. Will address all your points.

Regarding Reddit, we have our own custom handler for Reddit URLs which uses the Reddit API, which we are billed for when we exceed free limits.

For Terms of Service, you're right, that is definitely an oversight on our part. We just published both our Terms of Service and Privacy Policy on the website.

When it comes to comparing with GPT-5 and Claude, we do believe that our prompting, agent orchestration, and other core parts of the product such as parallel search results analysis and parallel agents are improvements on just GPT-5 and Claude, while also allowing it to run at much cheaper costs on significantly smaller models. Our v1 which we built months ago was essentially the same as what GPT-5 thinking with web search currently does, and we've since made the explicit choice to focus on data quality, user controllability, and cost efficiency over latency. So while yes, it might give faster results and work better for smaller datasets, both we and our users have found Webhound to work better for siloed sources and larger datasets.

Regarding account deletion, that is also a fair point. So far we've had people email us when they want their account deleted, but we will add account deletion ASAP.

Criticism like this helps us continue to hold ourselves to a high standard, so thanks for taking the time to write it up.

Could you share the session url via the feedback form if you still have access to it?

That's really strange, it sounds like Webhound for some reason deleted the schema after extraction ended, so although your data should still be tied to the session it just isn't being displayed. Definitely not the expected behavior.

Accuracy-wise we think it's almost there but probably still a few iterations away from being perfect. It's great at eliminating a lot of the collection time though.

Interestingly, we're working with B2B clients right now where we use Webhound to curate and then act as the "validation" layer ourselves. The agent lets us offer these datasets way cheaper with live updates, but still with human oversight.

Thanks for testing it! That's definitely a miss, sounds like it got confused about what you were looking for and went after board member pages instead of the actual meeting/document sites.

We're working on better query interpretation, but in the meantime you could try being more specific like "find BoardDocs or meeting document websites for each district" to guide it better. Also, you can usually figure out how it interpreted your request by looking at the entity criteria, those are all the criteria a piece of data needs to meet to make it in the set.

Yes, translation layer would probably be better terminology.

Did this resolve itself? If not shoot us an email at [email protected] and we can get it figured out.

Supabase

Thanks, fixed!

Fair point, most of our users have come from referrals/word of mouth so it hasn't really been an issue for us, but you're probably right that we should have more information on the landing page

We have, 2.5 Flash is about as small as we've been able to go while still delivering consistent results.