K0balt
If you want 1KW reliably off grid (24/7 with 24kwh battery) (let’s assume 5 hrs solar insulation) you’ll need around 20kw of panels. Even then, you’ll have occasional shortages in inclement weather.
If you are on the grid (theoretically unlimited battery, just averaging over the year) you’ll need 4.8 kw of panels.
Batteries are the hard part of solar.
AI small enough to run locally on a 30 dollar smartphone is far enough from useful to be effectively null.
Not only that, but AI is categorically terrible at ux review. I don’t mean aesthetics, I mean attention flow, workflow in general and what the user anticipates as a logical path to feature discovery and exploration. Increasingly, features are hidden behind workflows that meet a single use case but fail to be universally logical or in many cases, even accessible at all outside of the test case.
So, people… use AI in code review. It’s actually pretty great at it if you do it right, and you can ship better software with fewer bugs than ever before. But please, don’t pressure your ux guy to “10x” with AI. In fact, now that you are “10x”ing your development cycle, you are probably going to need to hire -more- ux developers if you don’t want your product to feel like slop even if it’s gold under the hood.
Just putting 100 of the smartest people in the planet in the same room doesn’t mean profit. It’s the underwear gnome problem. AI is no different in that regard.
I expect I’ll be able to run 1/2 terabyte models in 5 years for a couple $K in hardware. Inference is going to be a lot like Bitcoin mining. Only the peak of the power/performance curve is going to be profitable. Which means a glut of surplus hardware that can’t make money but makes tons of tokens.
What I meant though wasn’t agents building UX. I meant that in our case at least, the agent is the UX. “put a new extension 117 for Sam at the port next to Jane’s, and have it ring through to his cell when he’s not in the office. Leave the old 117 as is but only ring where Sam is, if that is known” basically, agents just do everything with SSH, and you can talk to them instead of poking around with a mouse. We don’t build ux for internal systems anymore. Local inference is strong enough to be reliable now with the correct tools and policies, and for 10-20k in hardware you are self contained. (Obviously $more if you are going to have a lot of people using it at the same time)
The “one wierd trick” is to build testing jugs for everything. Before a configuration is deployed, it’s tested against a toy instance, all existing functions are tested in all common cases, new functions verified, inputs fuzzed, adversarily abused, and stress tested against dummy loads. If its code, it gets examined against development policy, self containment, explicit interface walls, race conditions, code standards, complexity, factorability, and often A/B testing of different architectural choices. Except cryptography and a couple of other things, we generally prohibit the use of any libraries, it’s just not needed and the code never passes our tests anyway.
People are just YOLOing it and assuming that if it basically functions and looks right then it must be perfect. You’d have the same luck if you told your UX guy “test only the obvious, and if it looks about right at first glance, merge it. Don’t do extensive testing.”
The real thing with AI is that now you can afford to do a ridiculous amount of automated testing, setting up test fixtures, hardware simulators, mannequins of all the external interfaces, fuzzing, etc. Our testing code is 2-3x our product code, and we are shipping 10% of the bugs we shipped with humans, and it’s still 1/4 the cost and 4-10x the speed.
People shipping all the crap that is everywhere now have bad process engineering. Plain and simple. If you try to build software with AI like you used to with people, you’re going to have a bad time. The pipeline is very different.
The whole process hinges on one thing. You have to specify your code. Specification is code at its highest level of abstraction. So you are writing software by specification. The LLM pipeline is compiling it into code.
LLMs are useful in a back and forth to create the specification, but that’s where the human work is.
You don’t write software with a prompt. You specify an application, a user story, the protocols, and the API, then the LLM turns that into good code in a highly iterative and automated process. There is some human input needed during compilation, but not much.
The review pipeline is automated. Human review is rare.
The whole process hinges on one thing. You have to specify your code. Specification is code at its highest level of abstraction. So you are writing software by specification. The LLM pipeline is compiling it into code.
LLMs are useful in a back and forth to create the specification, but that’s where the human work is.
You don’t write software with a prompt. You specify an application, a user story, the protocols, and the API, then the LLM turns that into good code in a highly iterative and automated process. There is some human input needed during compilation, but not much.
Right now I’m running 2x 3090 and 128 gb ram on 3.8 flash-next180b (the quant is 97gb) and it hums at120t/s. It’s really a surprisingly good model for general tasks, and I’m soon to see how it does with more agentic kit. For agentic work it does slow down quite a bit when the context fills up… but what do you want for running a 180b model on consumer hardware?
Right now I have it set up as an administrator for our asterisk PBX and radius server, and it does higher level judgment on our cctv footage for notifications to security. It’s been really good so far. Agents are the new UI, everything just works over SSH now.
The difference is testing. Testing at every level. If crap code can’t move down the pipeline, you don’t get crap code. If a company is used to YOLOing code out the door with human developers and they try the same thing with LLMs they are going to have a bad day.
It’s basically an office of persistent agents with different defined roles and goals. The gatekeepers whole job is to refuse code. The engineers whole job is to write code that will pass. Each significant component gets its own engineer agent. There is an office manager that coordinates the agents, maintains the project documentation, keeps up the job boards, maintains scripts to keep agents pointed at the relevant onboarding documents As they come back from compaction, and manages office automation in general. Ther is an agent in charge of making sure the test coverage is as close to fully tested as possible, including refusing code that is not properly testable.
If it’s a big project, there may be an information management agent that manages the technical side of the documentation and maintains a searchable library, like an oracle. I may put in a personal assistant to help me to prioritize my involvement. Many projects also include an agent that creates mockups, jigs, and simulators so that tests can be run on code that needs a context.
I maintain another office just for research, because research can seriously sidetrack engineering. It has all of the standard staffing, but its whole job is to do research and write reports for other teams.
Agent persistence is maintained by having agents write down memories learnings and musings(not work related) and curiosities for the next instance to read in the onboarding for their seat.
Ethos, standards, practices, and policies are also in the onboarding process.
Every bit of code gets 10-50 review passes against extensive smells, rules, best practices, etc. Naive reviews and documentation driven development is your friend. If your code isn’t 100percent testable (even code previously considered untestable, because now you can create simulators and jigs) and you aren’t spending 5x+ tokens on review over coding, you’re going to have a bad time.
It’s still 5-10x less expensive than people, and you end up with higher quality code in the end.
We create higher quality, more testable code, with less complexity and less LOC per functional unit than we ever did with humans. By a long stretch.
Documentation driven development with constant naive review, delta review, documentation/code coherence review agents, test coverage review agents, development standards and architecture review, etc. We write code once, then review and revise each module from a variety of different paradigms until every line, every construct, every function, every interface is optimally constructed and documented. Agents learn to write code that passes with less flags by having it pushed back to them until it passes, and revising their learnings files.
By the time code makes it to a commit it’s been brought into compliance from a variety of perspectives and tested as completely as possible, along with mutants that test the tests. By the time it gets to a merge gate, most code has been through at least 50 reviews. Then we periodically sweep the entire codebase for cruft, vulnerabilities, corner cases, etc. If we find smells or clever sections, they are eradicated.
In total, the hot paths and critical infrastructure sees hundreds of reviews against thousands of rules and best practices.
Sure, it’s expensive, but still 1/4 the cost of regular development, and much, much fewer bugs.
Unfortunately, most people are out there just vibin, even at the big corps, and it’s a dumpster fire.
It’s easy to produce garbage software, but it’s not mandatory.
FWIW, well designed study probably would still support his premise. He at least needed a group with no help, a group with no help but books and a calculator (tools), a group with human assistance, and a group with AI assistance, and a group that does an unrelated activity during the priming period.