payneio

And by "wisdom of the croud", I'm referring to sharing what works well and what doesn't and building good approaches into the frameworks... encoding human expertise. We do it all the time.

Compiling and evaluating output are types of fact checking. We've done more extensive automated evaluations of "groundedness" by extra ting factual statements and seeing whether or not they are based on input data or hallucinated. There are many techniques that work well.

For comparisons, you can ask the model to eval on various axis e.g. reliability, maintainability, cyclometeic complexity, API consistency, whatever, and they generally do fine.

We run multi-trial evals with multiple inputs across multiple semantic and deterministic metrics to create statistical scores we use for comparisons... basically creating benchmark suites by hand or generated. This also does well for guiding development.

I feel that. I've been on an emotional roller-coaster for three years now. I didn't expect any of this before then. :O

Ah! Gotcha. Thanks for the clarification.

The use cases I'm thinking that require cloud architecture are scaling up with GPUs (for self-hosted intelligence workloads). Also, Wild Cloud is meant to meet community needs more than individual needs (thought it will do that, too) so I'm imagining needing to scale horizontally more than just vertically. I would still recommend putting things like home-assistant or a home media server on SBCs.

It still is way more complex than I want it to be for a person to set up a local cluster, but I'm still hopeful I can make it simpler.

Also... "scammer and AI grifter"?? Damn dude. It's any early-stage open-source experiment result and, mostly, just talking about how it makes me question whether or not I'll be programming in the future. Nobody's asking for your money.

I get it. I've been through cycles of this over the past three years, too. Used a lot of various tools, had a lot of disappointment, wasted a lot of time and money.

But this is the kinda the whole point of my post...

In our system, we added fact checking itself, comparing different approaches, summarizing and effectively utilizing the "wisdom of the crowd" (and it's success over time).

And it made it work massively better for even non-trivial applications.

Thanks for the extract. I feel quite comfortable that my post is on-topic and gratifying. I understand others may disagree (and do in nearly every post on HN)

So, what we do is automate the hand-holding. In your physics simulation example, you can have the system attempt to compile on every change and fix any errors it finds (we use strict linting, type-checking, compile errors, etc.); and you can provide a metric of "good" and have it check for that and revise/iterate as needed. What we've found particularly useful is breaking the problem into smaller pieces--"The Unix Philosophy" as the system is quite capable of extracting, composing, defining APIs, etc. over small pieces. Make larger things out of reliable smaller things like any reasonable architecture.

These things are not "creative"... they are just piecing together decent infrastructure and giving the "actor" the ability to use it.

Then break planning, design, implementation, testing, etc. apart and do the same for each phase--reduce "creativity" to process and the systems can follow the process quite nicely with minimal intervention.

Then, any time you do need to intervene, use the system to help you automate the next thing so you don't have to intervene in the same way again next time.

This is what we've been doing for months and it's working well.

How are you verifying your claims? I'm actually seeing results that you describe as being impossible.

Not just you. A lot of people think that, I'm sure.

Not sure what you mean about the organizational abstractions. FWIW, I've worked in five startups (sold one), two innovation labs, and a few corporations for a few years. I feel like I've seen our industry from a lot of different perspectives and am not sure how you imagine being at Microsoft for the past 5 years would warp my brain exactly.

It's not, actually. It's a glimpse into a research project being built openly and made freely, by the engineers building it, to anyone who wants to take a look.

The products will come months from now and will be introduced by the marketing team.

Yes. These are all the same points I used to believe until recently... in fact the article I write two months earlier was all about LLMs not being able to think like us. I still haven't squared how I can believe both things at the same time. The point of my article was to try to explain why I think otherwise now. Responding to your thoughts in sequence:

- These systems can re-abstract and decompose things just fine. If you want to make it resilient or scalable it will follow whatever patterns you want to give it. These patterns are well known and are definitely in the training data for these models.

- I didn't jump to the conclusion that doing small things will make anything possible. I listed a series of discoveries/innovations/patterns/whatever that we've worked on over the past two years to increase the scale of the programs that can be generated/worked-on with these systems. The point is I'm now seeing them work on systems at the level of what I would generally write at a startup, open source project, or enterprise software. I'm sure we'll get some metrics soon on how functional these are for something like Windows, which, I believe is literally the world's single largest code base.

- "creativity" and novel-seeking functions can be added to the system. I gave a recent example in my post about how I asked it to write three different approaches to integrate two code bases. In the old world this would look like handing a project off to three different developers and seeing what they came up with. You can just brush this all of with "their just knowledge bases" but then you have to explain how a knowledge base can write software that would take a human engineer a month on command. We have developed the principle "hard to do, easy to review" that helps with this, too. Give the LLM-system a task that would be tedious for a human and then make the results easy for a human to review. This allows forward progress to be made on a task at a much-accelerated pace. Finally, my post was about programming... how much creativity do you generally see in most programming teams where they take a set of requirements from the PM and the engineering manager and turn that into a code on a framework that's been handed to them. Or take the analogy back in time... how much creativity is still exhibited in assembly compilers? Once creativity has been injected into the system, it's there. Most of the work is just in implementing the decisions.

- You hit the point that I was trying to make... and what sets something like Amplifier apart from something like Claude Code. You have to do MUCH less prompting. You can just give it an app and tell it to improve it, fix bugs, and add new features based on usage metrics. We've been doing these things for months. Your assertion that "we would have already replaced ALL programmers" is the logical next conclusion... which is why I wrote the post. Take it from someone who has been developing these systems for close to three years now... it's coming. Amplifier will not be the thing that does this... but it shows techniques and patterns that have solved the "risky" parts enough to show the products will be coming.

Yes. Please read it. I'm looking for collaborators. The links in this article point to recent work on Wild Cloud so you can see where it's currently at.

Wild Cloud will is a network appliance that will let you set up a k8s cluster of Talos machines and deploy apps curated apps to it. It's meant to make self-hosting more accessible, which, yes, I think can help solve a lot of data sovereignty issues.

I'm not sure what you mean by "barely programs"

What's wrong with "self promotion"? The point of this space has always been promoting projects. That's what Y Combinator is all about

Yes, I code a lot. My GitHub is public as are many of the projects I work on.

FWIW, finished an eval of claude code against various tasks that amplifier works well on:

The agent demonstrated strong architectural and organizational capabilities but suffered from critical implementation gaps across all three analyzed tasks. The primary pattern observed is a "scaffold without substance" failure mode, where the agent produces well-structured, well-documented code frameworks that either don't work at all or produce placeholder outputs instead of real functionality. Of the three tasks analyzed, two failed due to placeholder/mock implementations (Cross-Repo Improvement Tool, Email Drafting Tool), and one failed due to insufficient verification of factual claims (GDPVAL Extraction). The common thread is a lack of validation and testing before delivery, combined with a tendency to prioritize architecture over functional implementation.

Here's a writeup of the project for more context: https://paradox921.medium.com/amplifier-notes-from-an-experi...

I've tried it. It works better than raw Claude. We're working on benchmarks now. But... it's a moving target as amplifier (an experimental project) is evolving rapidly.

Hey all! I'm one of a handful of developers on this project. Great to see it's getting some interest!

For context, we are right in the middle of building this thing... multiple rebuilds daily since we are using it to build itself. The value isn't in the code itself, yet, but in the approaches (UNIX philosophy, meta-cognitive recipes, etc.)

We are really excited about how productive these approaches are even in this early stage. We are able to have amplifier go off make significant progress unattended for sometimes hours at a time. This, of course, raises a lot of questions on how software will be built in the near future... questions which we are leaning into.

Most of our team's projects, unless they have some unresolved IP or are using internal-only systems, are built in the open. This is a research project at this stage. We recognize this approach it too expensive and too hacky for most independent developers (we're spending thousands of dollars daily on tokens). But once the patterns are identified, we expect we'll all find ways to make them more accessible.

The whole point of this is to experiment and learn fast.

Thanks for pushing for a realignment of product expectations. I agree.

I wrote up some details of investigations of a chatbot we created in Microsoft Research using a technique of creating synthetic memories with LLMs and RAG.

Key takeaway is that it produced a bot that was perceived as more empathetic than ChatGPT with GPT-4 (and other chatbots) and allowed context to be maintained for extremely long chat conversations.

Too true.

Linux plz