Sure it's busywork. But it's a lot of busywork very fast.
I'm really curious what your problem domain is, like specifically what sort of code are you asking it to change and what changes are you asking for.
I just gave o1 and Sonnet a total layup question (optimization that had a huge win simply by filtering an array before sorting it vs the other way around) and neither model got the solution right, both of them came up with ~hundred lines of code, neither model's code worked on the first try. It took me like 10 minutes to refactor and optimize the code for a 6x speedup and it would take longer than that to debug the AI code to even make it run. (I spent 10 minutes prompting/editing to try to get the generated solutions to run)
Also the initial code was 11 sloc, my solution is 14 sloc, and claude was 70 sloc and o1 was 93. idfk, i just don't think we're there yet
Obviously nothing complicated but it takes non-zero time. It did it in one shot in about 30s. Didn’t have to look at and docs (I don’t have the yaml lib memorized). Got the python typing right which would have been a bit of a pain. A lot faster than doing it myself, even with reviews. Tests were solid so I could tell it worked.
The filter example you give seems like they should have aced it. Not sure what went wrong but it has easily done work like that for me. I usually am half way through typing the method name when the rest autocompletes. Are you using a tool with good context management like cursor?
Open Weights models has it's place (in training custom agents and custom services), but if you are knowledge worker, using a model even 5% less than SOTA is extremely dumb
Also, in general I don't find the difference between SOTA models and local models to be that significant in the real world even when used in the exact same way.
Does this run with VSCode and how hard is it to set this up?
you'll then have to download a model, which ollama makes very easy. choosing which one will depend on your hardware but the biggest QwenCoder2.5 you can fit is a very solid starting place. it's not ready for your grandma, but it's easy enough that I'd trust a junior dev to be able to get it done
I just read the parent post, lol.
Also keep in mind that, even though it's "unified memory", the OS enforces a certain quota for the GPU. If I remember correctly, it's something like 2/3 of the overall RAM.
I'm glad it's open source so I was able to fix most of the issues I had with it and now my copy is in a great place. The documentation is in places many versions behind the actual code so it can be tough to figure out how to set things up when you're venturing off the beaten path. That all being said the granularity of control you have when using local models leads to an experience that's far better than Cursor/Copilot, I really enjoy that it reads my mind a lot of the time now (because I have prompt engineered it to know how I think).
Ultimately, isn't this just the way of things? https://thot-experiment.github.io/forever-problems/?set%20up...