gregwebs
Interesting thing is that when I asked a follow up question it answered but showed: This answer came from what the AI already knows, not the library, so there's no source to check. Ask “look that up” to get one.
When I asked "look that up" it was able to. It highlights relevant sections.
The information it was working off in my case was WikiHow and there aren't any citations in that information.
Running on a 32GB Mac M1 using Qwen 27B. Ollama is showing as taking 19GB steady state and 23GB while answering on Activity Monitor (sometimes a little more and sometimes a little less).
Its amazing this can be done, but it seems like someone should be able to develop a much smaller model that can be used reliably for this use case. There are smaller models available to use and download but it is stated that they have some reliability issues.
[1] https://github.com/gregwebs/skills-sdlc/blob/main/skills/rus... [2] https://github.com/gregwebs/skills-sdlc/blob/main/skills/ver...
This goes to show the benefit of maintaining separations between the control plane and the data plane.
This site does say it has SQL logs which have some amount of customer data and there is a mention of some snapshots/imports- this looks like a case where the control plane and data plane end up getting entangled. Its possible the attacker could have reset customer database passwords to try to connect to them or could have exfiltrated customer data by initiating snapshots but just didn't want to deal with the increased risk of detection.
TiDB is for large scale data. You generally wouldn't use it unless you have at least 1TB of data. A 10TB leak of information is less than the size of a single small customer database for them.
They are showing this new version scoring significantly higher across multiple coding benchmarks, beating Sol 5.6 (6.1 is the latest) across 10/11 benchmarks.
I am going back to DeepSeek for writing code. I was trying out using Sol and liking the result, but this should close a lot of that gap. I use Opus/Sol for an implementation plan and code review also use them for grilling. DeepSeek is also running verifications.
This technique seems dubious- what is stopping AI from doing the computer work of legal secretaries and court clerks as well?
Solving math problems is one of the hardest theoretical jobs. Software engineering is one of the harder practical jobs. AI is now much better at these tasks than humans because it has been able to properly train for them.
For other computer-based jobs it's a matter of AI getting properly trained for them. If a job is harder to verify or the training data is more limited, it may be much more difficult to train AI for it but I don't think there's any reason to think it won't happen eventually.
Originally at https://www.rd.com/article/how-honest-are-dentists/ it now seems available here: https://asset.cdocs.com/uploads/forum/attachments/258722/Rea...
I have found that agents are good at writing Verus proofs in a Rust code base, and that these proofs have only a minimal negative effect on the normal compilation path. I have a skill for using Verus: https://github.com/gregwebs/skills-sdlc/blob/main/skills/rus...
If you can direct AI and produce more value per $/time spent then a non software engineer directing AI to develop software then you still can produce a lot of value. This should be the case now for most things that are not a prototype. The job has changed from "writing code" to "shipping software". A technical lead.
Software products making ever more intelligent AI software development accessible will continue to reduce the need for software engineer involvement. But there’s also going to be 1000x more software. So there may continue to be opportunities for those that adapt.
This should be the case now for anything that is not a prototype. The job has changed from "writing code" to "shipping software". A technical lead.
As AI gets more intelligent and the tooling around it develops and organizations figure out how to get it access to the data, the level of involvement needed for software engineers will decrease. Since the cost of developing software is plummeting, the amount of software being used is going to expand greatly- that will keep the technical lead positions available longer and also create more opportunities for consultant engagements.
Although someone can use AI to rewrite every software dependency and every paid piece of software, its still not worthwhile for them for most software, especially if the price of the software is low compared to the complexity it hides.
Users of a paid software product still need a human to be responsible for it.
As you develop your skills with AI, you now have the ability to develop your own software products. So there's an opportunity to become a small business owner. You didn't want to do these non-technical roles, but now you can continue to direct AI to automate all the aspects of the business.
The job of a programmer is now to be a technical lead and work through technical decisions with AI, write specs, and review work. But as AI gains intelligence and organizations figure out how to give it access to the information it needs, it will make better technical decisions than humans.
As long as a programmer can in some way produce more value/$ using AI then someone that doesn't know programming, then there's a huge value to programmers. But I don't see the place where AI can't go up the chain and do that itself as it gains more intelligence.
This is effectively true of any job that can be done at a computer. Although programming is one of the more difficult jobs its also one that is easy to train on. My advice if one's main goal is job security would be to do something in the physical world.
DeepSeek is horrible at grilling sessions (the /grill* skills to make technical decisions). It doesn't know how to explain things. Maybe the skill could be adjusted. It also doesn't come up with as good solutions as Opus/Sol.
What I use it for is
* the orchestator of my coding workflows
* the tester/verifier of code changes
* the sub agent that explores code or does web searches
* putting together code base research reports
They are all working together in Pi using the extension @tintinweb/pi-subagents where my workflow skill is calling different subagents that use different models.
Luna is cost competitive, but doesn't score as well on intelligence. I do need the intelligence for most of what I use it for, so I am not motivated to use Luna. Haiku also doesn't seem like a competitive price/performance mix.
If you don't want to use their harness then you wouldn't use docker agent and instead use their `sbx` cli to run the harness of your choice (claude, codex, pi, etc).
I think it will be great if Docker can get people used to using secure VMs. I am developing a similar project (still a work in progress): https://github.com/gregwebs/agent-vm
TDD doesn't help the agent figure out what it is supposed to do now- it is already writing the code and now and it must know what it is supposed to do to write it.
I am trying to use evidence-based approaches. I am only observing better results and tweaking now, but I plan to benchmark when tweaking is done.
However, I found the same problems with agents that it has with humans (usually with humans it isn't TDD but code coverage requirements). The problem is the tests are now written to satisfy a bureaucracy rather than to properly verify the code. And there are real studies that point out the low value of these types of bureaucratic testing requirements. For example, when models are just required to write a test file, they don't produce a better result: https://arxiv.org/abs/2602.07900
I wrote a /verify skill that focuses on properly verifying code and I am finding it works a lot better. I do need to revise it now- in practice certain parts of the skill are doing all the heavy lifting and others are more dead weight. But it does seem to be properly orienting the agents towards finding defects. https://github.com/gregwebs/skills-sdlc/blob/main/skills/ver...
My thought though has always been that I don't want there to be agent-only designated documentation.
I use mattpocock/skills and that generates ADRs (Architectural Decision Records). That only uses skills, including a setup skill that will write a few pointers in AGENTS.md. I always have a CONTRIBUTING.md to document development flow and a CODING_STANDARDS.md. Between those and the README.md and architecture documentation and commit messages the agents seem to be able to find and use docs and keep them up to date. We are also writing a lot of specs and putting those in Github issues.
From looking at their Terminal-Bench graph, anything you would use level "high" or above for Sonnet it seems like you should consider using Opus instead.
OpenAI Luna is a lot cheaper. But DeepSeek seems smarter and the cost seems similar.
How you do this (and how deeply) I think is really the limit. I am doing this by focusing heavily on the design phase with grilling and trying to continually improve process to need less effort in the review phase. Are your models doing automated reviewing and testing before pushing out the PR (themselves)?
I think in the long run as models and the tools around them get better and cheaper, those that abdicate understanding will be able to achieve more. Although programmers think of that as irresponsible, ask yourself what does a tech lead do? And then what does a CTO do, etc?
One thing that I have found is that as the frontier models get better there is less need for agents with specialized personas. I actually don't don't use those anymore- I just use agents that have different models and reasoning levels. I have a generated CODING_STANDARDS.md document and a skill for architecture design and a skill for implementing testing [2] that are referenced by a single reviewer. I do implement a 2-pass review though [3].
I would be interested to know if you have found anything similar as models get better. It seems though that you are sharing a single exploration and then sharing the context across the specialized reviewers to dramatically reduce the cost of your approach. Does this have to be in the harness- that is if you write out the shared context to a file does that increase your costs a lot?
I also wonder how intensively are the models able to test their changes? The number one quality improvement I have found is not review but having the model properly test its code. I have a skill that is helping [4], but I also have to spend time to establish a pattern of testing with tools beyond just unit tests. The testing takes significant effort, and this is again where the cost savings of DeepSeek shine.
[1] https://github.com/mattpocock/skills/blob/main/skills/engineering/codebase-design/SKILL.md
[2] https://github.com/gregwebs/skills-sdlc/blob/main/skills/verify/SKILL.md
[3] https://github.com/mattpocock/skills/blob/main/skills/engineering/code-review/SKILL.md
[4] https://github.com/gregwebs/skills-sdlc/blob/main/skills/verify/SKILL.mdI do have an /implement-simple workflow to skip the planning phase, but even that doesn't skip the review.
Are you doing your own intensive reviews of the model code? Can you share the prompts you are using as I have?
My bar for what models produce without human intervention is much lower defect than what a human would produce. The human interaction is mostly to guide the design and then the review burden is very low. I suspect your bar for what agents produce is lower- you are taking more of the review burden. I also suspect that you are measuring time more than actual cost since your employer is paying and that you are comparing to Sonnet rather than DeepSeek (DeepSeek 4.1 again is 20-40x cheaper than Sonnet). You mention hundreds of millions of tokens (my reviews don't use that much), but even that costs ~$1 on the DeepSeek side.
I think you are taking exactly the right approach at your employer given the cost is free and you only have access to Anthropic models.
I also agree that its a big mistake to have a flash model implement without a strong model reviewing.
I have Opus plan, Deepseek implement the code, and then review with Opus [1]. In this workflow I am saving a lot of money by having Deepseek do the implementation. Note that the review back-and-forth is fully automated [2], so it doesn't take any extra attention from me.
[1] https://github.com/gregwebs/skills-sdlc/tree/main/skills/implement
[2] https://github.com/gregwebs/skills-sdlc/blob/main/skills/code-review-with-followup/SKILL.mdThe problem is that the student is in charge of the tutor and often tells them to just do their homework for them and not check whether they have learned anything.
LLM providers should formalize a tutoring mode. We should have prompts available to create this mode. This should be the default way students use LLMs. LLMs may not be good enough to do this now, but I think there is already enough raw intelligence and its a matter of training models to be good tutors, to be experts and specific subjects, and developing software systems for tutoring.
In this model there are few lectures. This is more similar to Montessori but with on demand tutoring. The teacher provides objectives and materials for the students and tutors. The tutoring process can be done during class for teachers to ensure it is working well. There can be larger blocks of office hours for students to do their tutoring during.
As AI gets more intelligent, it will also be capable of doing all the grading. This includes oral exams. Teachers need to proctor the exams in person.
Anyone will be able to tutor themselves on any subject. Proctored examinations to certify knowledge do not need to be very expensive. The value of the school environment will not be to transfer knowledge from a teacher but to have peers working on the same thing and teachers and an environment invested in training the next generation.
There are going to be cases where you want to white list an org or a repo for read access that is not under your control and Github filtering will be a simple way to do that.
Github can be a source of hostile code, prompt injections, and exfiltration- you may want to lock down using repos that aren't yours.
The tool inherited this feature from the prior implementation and its something I am still exploring.