cleverwebble
The naive solution I could come up with would be really expensive with openai, but if you have an open source model, you can write up custom inference that goes one-token-at-a-time through the text, and on each token you look up the difference in logprobs between the token that the LLM predicted vs what was actually there, and use that to color the token.
The downside I imagine to this approach is it would probably tend to highlight the beginning of bad code, and not the entire block - because once you commit to a mistake, the model will generally roll with it - ie, a 'hallucination' - so logprobs of tokens after the bug happened might only be slightly higher than normal.
Another option might be to use a diffusion based model, adding some noise to the input and having it iterate a few times through, then measuring the parts of the text that changed the most. I have only a light theory understanding of these models though, so I'm not sure how well that would work
I've spent the last few months using Claude Code and Cursor - experimenting with both. For simple tasks, both are pretty good (like identifying a bug given console output) - but when it comes to making a big change, like adding a brand new feature to existing code that requires changes to lots of files, writing tests, etc - it often will make at least a few mistakes I catch on review, and then prompting the model to fix those mistakes often causes it to fix things in strange ways.
A few days ago, I had a bug I just couldn't figure out. I prompted Claude to diagnose and fix the issue - but after 5 minutes or so of it trying out different ideas, rerunning the test, and getting stuck just like I did - it just turned off the test and called it complete. If I wasn't watching what it was doing, I could have missed that it did that and deployed bad code.
The last week or so, I've totally switched from relying on prompting to just writing the code myself and using tab complete to autocomplete like 80% of it. It is slower, but I have more control and honestly, it's much more enjoyable of an experience.
We'd go to Dollywood a few times a year - she would give out free tickets to people who worked in Gatlinburg to go. It's really well run, and their water park is great too. Growing up, we'd ride the train when we visited. While I didn't appreciate it much as a kid, when I grew up I realized how awesome of an opportunity that was.. I moved away from Tennessee about 12 years ago, one of the biggest things I have missed is Dollywood and their big steam train.
This really only works though if:
1) The task is limited to a relatively small domain (relatively small could probably be misnomer, as most LLMs are trying to solve every-problem-all-at-once. As long as you are having it specialize in a specific field even, FT can help you achieve superior results.) 2) You have high quality examples (you don't need a lot, maybe 200 at most) Quality is often better than quantity here.
Often, distillation is all you need. Eg, do some prompt engineering on a high quality model (GPT-4.1, Gemini-Pro, Claude, etc.) - generate a few hundred examples, optionally (ideally) check for correctness via evaluations, and then fine tune a smaller, cheaper model. The new fine tuned model will not perform as well at generalist tasks as before, but it will be much more accurate at your specific domain, which is what most businesses care about.
You can be direct and respectful, but this was not respectful, this was just aggressive.
If the US government didn't already have this kind of tech, they would spend millions just for the same prototype they built. And probably tens or hundreds of millions for a final product.
I think that's a story
Edit: now that I'm thinking about it, I don't think I worked at a tech company where people were fired/asked to resign more than 2.5x than those that resigned on their own.
Internally, Indeed watches a ton of metrics, but our primary metric is how many people we ultimately helped get a job. We rarely release a feature unless we can prove it, with hard data, that it does. Secondary metrics monitor things like jobseeker and employer efficiency (how easy is it to find a mutual match.) In the last two years, the pricing model, the recommendation engine, everything is completely different.
I know I'm being pessimistic, but the CEO isn't the one having to spend the next few months trying to find a good job when everyone is on a hiring freeze.
If you've seen the question/answer before just say so! I will totally appreciate the honesty and it goes a long way.