grbsh

The fundamental frustration most engineers have with AI coding is that they are used to the act of _writing_ code being expensive, and the accumulation of _understanding_ happening for free during the former. AI makes the code free, but the understanding part is just as expensive as it always was (although, maybe the 'research' technique can help here).

But let's assume you're much better than average at understanding code by reviewing it -- you have another frustrating experience to get through with AI. Pre-AI, let's say 4 days of the week are spend writing new code, while 1 day is spent fixing unforseen issues (perhaps incorrect assumption) that came up after production integration or showing things to real users. Post-AI, someone might be able to write those 4 days worth of code in 1 day, but making decisions about unexpected issues after integration doesn't get compressed -- that still takes 1 day.

So post-AI, your time switches almost entirely from the fun, creative act of writing code to the more frustrating experience of figuring out what's wrong with a lot of code that is almost correct. But you're way ahead -- you've tested your assumptions much faster, but unfortunately that means nearly all of your time will now be spent in a state of feeling dumb and trying to figure out why your assumptions are wrong. If your assumptions were right, you'd just move forward without noticing.

Why not just use Claude by itself? Opus and Sonnet are great at producing pixel coordinates and tool usages from screenshots of UIs. Curious as to what your framework gives me over the plain base model.

I recently got the Pavlok device to stop a bad habit - compulsive nail biting. It worked extremely well! It works less well for thinks like distraction, because you have to manually trigger the negative stimulus.

I find the prospect of an AI continuously monitoring and course correcting very interesting. Like GPS/ maps -- if you go off course, it re-routes you. What will people build with real-time AI like this in the future? I think Cluely may be the best example of this.

I'd personally like a version of the Pavlok device that has a built in camera / mic that allows me to configure these type of triggers with real-time monitoring. It'll probably be a while until we can run an LLM on a watch-sized device in 1-3s but still have ~8hr battery life.

Knowledgework is AI that can actually save you time, make you better organized, and do better work. Knowledgework is the answer to question, “Why isn’t ChatGPT more useful for real-world work tasks?”

While ChatGPT is a better Google search, Knowledgework is like asking a copy of your own brain for help. Today’s chat interfaces are flawed because they require the user to micromanage: to not only know exactly what can and should be done, but to describe in detail all the specifics required to do it. Knowledgework fixes this with two elements that I believe create a new paradigm for AI enabled software: proactivity and omniscience. Without these two properties, you can’t delegate meaningful real world tasks to AI.

ChatGPT isn’t useful for this. Not because it’s not intelligent enough (it is), but because it lacks the right knowledge. When I speak to knowledge workers about how they use AI for work, they tell me they’d like to delegate more tasks to AI, but that they experience this frustration of needing to micromanage it, re-teaching everything about their project and their team every time.

How does Knowledgework solve omniscience and proactivity to create an AI assistant that’s useful for delegation of real-world work? It’s a desktop vision AI that watches you work, like an intern who is shadowing you, or a pair programmer. It learns the rich, internal (often unwritten) knowledge specific to your projects that's required to contextualize them. It organizes this into neat, understandable documentation of everything you’re doing: a hyperlinked wiki that connects all of your team’s concepts, decisions, definitions, acronyms, tools, etc.

This explicit representation of your knowledge powers the AI assistant to enable useful delegation — but it’s also really useful in it of itself. Curious as to how your team came to a decision on something? Click through the wiki to get context.

The other main feature is the Timeline. It’s kind of like a log or an objective summary of how you spent your time. While the wiki mirrors humans’ associative and conceptual memory, the timeline represents episodic memory. This enables you to visually search through your time: imagine you remember solving a similar problem a few weeks ago, but you don’t quite remember when. By going through the Timeline, you can quickly scan to find the specific work session and ask about what you did.

Together, these representations of your knowledge and experience along with the AI assistant running on top begin to feel like a sort of “digital second brain”. Since I started using it, I’ve had the experience where I’m hesitant to even do things on other devices, because it feels like anything I do there is ephemeral.

If you’re excited to upload your mind and see what the future looks like with this tech, sign up for the waitlist here: https://knowledgework.ai.

Ooh, I really like the idea about deciding whether to use the big or small model based on task specificity.

I know moondream is cheap / fast and can run locally, but is it good enough? In my experience testing things like Computer Use, anything but the large LLMs has been so unreliable as to be unworkable. But maybe you guys are doing something special to make it work well in concert?

Why would you ever hire a human to perform some task for you in a company? They're known for having problems with ambiguity and precision in communication.

Humans require a lot of back and forth effort for "alignment" with regular "syncs" and "iterations" and "I'll get that to you by EOD". If you approach the potential of natural interfaces with expectations that frame them the same way as 2000s era software, you'll fail to be creative about new ways humans interact with these systems in the future.

It's a great point that this is how we primarily used to interact with businesses and services, but we've moved on. For Gen-Z, e.g., many will refuse to use the product or service if they have to speak to an actual human. Just like we're now not willing to take boat across the ocean for 3 months, but before airplanes this was not uncommon.

I think we can have the best of both worlds here. We want the precision and speed of using vi commands, but we want the discoverability of GUI document editors. LLMs may be able to solve the discoverability problem. If the editor can be highly confident that you want to use a given a command, for example, it can give you an intellisense like completion option. I don't think we've cracked the code on how this UX should work yet though -- as evidenced by how many people find cursor/copilot autocompletion suggestions so frustrating.

The other great thing about this mode is that it can double as a teaching methodology. If I have a complicated interface that is not very discoverable, it may be hard to sell potential users on the time investment required to learn everything. Why would I want to invest hours into learning non-transferrable knowledge when I'm not even sure I want to go with this option versus a competitor? It will be a far better experience if I can first vibe-use the product , and if it's right for me, I'll probably be incented to learn the inner workings of it as I try to do more and more.

Tesla / Waymo is a perfect illustration of the point, but the Bitter Lesson doesn’t allow us to pick a winner here. The Bitter Lesson tells us that the Tesla approach (fully end to end, minimizing hand coded features / logic) will _ultimately_ win out. The Bitter Lesson does not tell us that this approach has to economically justify itself 1 year in, 5 years in, or that the approach when the technology is immature will allow a company to avoid bankrupting itself in the meantime while they wait for the data and compute to scale.

In other words, just because we know that ultimately (possibly in 20+ years) the Tesla compute-only approach will be simpler and more effective, Tesla might not survive to see this happen. Instead, manual feature engineering and hacking can always give temporary gains over data and compute driven approaches. The bitter lesson was clear about this. I suspect Waymo will win, and at some point in the future once they are out of their growth at all costs stage, they will transition into their maximum value extraction stage, in which vision will make significantly more economic sense than LiDAR. But once they win, they’ll have plenty of time to see the bitter lesson through its ultimate consequences. Elon is right, but he’s probably too early.

“The vast majority of our work is already automated to the point where most non-manual workers are paid for the formulation of problems, social alignment in their solutions, ownership of decision making / risk, action under risk, and so on”

Exactly! What a perfect formulation of the problem.

How will this be gamed for neo-seo spam?

Is this more than just ChatGPT with search API resulted concatenated the prompt?

It feels like it might be. It feels tasteful in the same way that Apple ecosystem integrations just work really nicely and intuitively. But then again, there is an art to keying and retrieving embeddings, and it might just be that.

I think this is a bit short sighted, but I’m not sure how short. I suspect in the future, code will be something in between what it is today, and a build artifact. Do you have to maintain bytecode?

I think there is something deeper going on: “coding” is actually 2 activities: the act of implementing a solution, and the act of discovering the solution itself. Most programmers are used to doing both at once. But to code effectively with an LLM, you need to have already discovered the solution before you attempt to implement it!

I’ve found this to be the difference between writing 50+ prompts / back and for the to get something useful, and when I can get something useful in 1-3 prompts. If you look at Simon’s post, you’ll see that these are all self-contained tools, whose entire scope has been constrained from the outset of the project.

When you go into a large codebase and have to change some behavior, 1) you usually don’t have the detailed solution articulated in your mind before looking at the codebase. 2) That “solution” likely consists of a large number of small decisions / judgements. It’s fundamentally difficult to encode a large number of nuanced details in a concise prompt, making it not worth it to use LLMs.

On the other hand, I built this tool: https://github.com/gr-b/jsonltui that I now use every day almost entirely using Claude. “CLI tool to visualize JSONL with textual interface, localizing parsing errors” almost fully qualifies this. In contrast, my last 8 line PR at my company, while it would appear much simpler on the surface level, contains many more decisions, not just of my own, but reflecting team conversations and expectations that are not written down anywhere. To communicate this shared implicit context with Claude would be so much more difficult than to perform the change myself.

> limits of the human mind when more and more AI-generated code will be used

We already have a technology which scales infinitely with the human mind: abstraction and composition of those abstractions into other abstractions.

Until now, we’ve focused on getting AI to produce correct code. Now that this is beginning to be successful, I think a necessary next step for it to be useful is to ensure it produces well-abstracted and clean code (such that it scales infinitely)

I feel the exact same! I built this tool to make it much easier for me to bring LLMs into existing workflows: https://github.com/gr-b/repogather

It helps find relevant content to copy to your clipboard (or just copies all files in the repo, with exclusions like gitignore attended to) so you can paste everything into Claude. With the large context sizes, I’ve found that I get way better answers / code edits by dumping as much context as possible (and just starting a new chat with each question).

It’s funny, Anthropic is surely losing money on me from this, and I use gpt-mini via api to compute the relevancy ratings, so OpenAI is making money off me, despite having (in my opinion) an inferior coding LLM / UI.