Like the the other day, I gave it a bunch of use cases to write tests for, the use cases were correct the code was not, it saw one of the tests broken so it sought to rewrite the test. You risking suboptimal results when an agent is dictating its own success criteria.
At one point I did try and use seperate Claude instances to write tests, then I'd get the other instance to write the implementation unaware of the tests. But it's a bit to much setup.
Additionally, if their code has broken existing tests, it "fixes" them by not fixing the code under test, but changing the tests... (assert status == 200 becomes 500 and deleting code.)
Tests "pass." PR is opened. Reviewers wade through slop...
Sometimes it just add tests that lock in specific quirks of the code that weren’t necessarily intentional
It’s one of those things which has always strikes me funny about programming is how less usually really is more
Hard agree!
For one off, this is fine. For anything maintainable, that needs to survive the realities of time, this is truly terrible.
Related, my friend works in a performance critical space. He can't use abstractions, because the direct, bare metal, "exact fit" implementation will perform best. They can't really add features, because it'll throw the timing of others things off to much, so usually have to re-architect. But, that's the reality of their problem space.
Historically pure machine code with jumps etc lacked any from of encapsulation as any data can be accessed and updated by anything.
However, you would still use abstractions. If you pretend the train is actually going 80.2 MPH instead of somewhere between 80.1573 MPH to 80.2485 MPH which you got from different sensors you don’t need to do every calculation that follows twice.
> In software, an abstraction provides access while hiding details that otherwise might make access more challenging
I read this as "an encapsulation of a concept". In software, I think it can be simplified to "named lists of operations".
> Historically pure machine code with jumps etc lacked any from of encapsulation as any data can be accessed and updated by anything.
Not practically, by any stretch of the imagination. And, if the intent is to write silly code, modern languages don't really change much, it's just the number of operations in the named lists will be longer.
You would use calls and returns (or just jumps if not supported), and then name and reference the resulting subroutine in your assembler or with a comment (so you could reference it as "call 0x23423 // multiply R1 and R2"), to encapsulate the concept. If those weren't supported, you would use named macros [2]. Your assembler would used named operations, sometimes expanding to multiple opcodes, with each opcode having a conceptually relevant name in the manual, which abstracted a logic circuit made with named logic gates, consisting of named switches, that shuffled around named charge carriers. Say your code just did a few operations, the named abstraction for the list of operations (which all these things are) there would be "blink_light.asm".
> If you pretend the train is actually going 80.2 MPH instead of somewhere between 80.1573 MPH to 80.2485 MPH which you got from different sensors you don’t need to do every calculation that follows twice.
I don't see this as an abstraction as much as a simple engineering compromise (of accuracy) dictated by constraint (CPU time/solenoid wear/whatever), because you're not hiding complexity as much as ignoring it.
I see what you're saying, and you're probably right, but I see the concepts as equivalent. I see an abstraction as a functional encapsulation of a concept. An encapsulation, if not nonsense, will be some meaningful abstraction (or a renaming of one).
I'm genuinely interested in an example of an encapsulation that isn't an abstraction, and an abstraction that isn't a conceptual encapsulation, to right my perspective! I can't think of any.
[1] https://en.wikipedia.org/wiki/Abstraction_(computer_science)
[2] https://www.tutorialspoint.com/assembly_programming/assembly...
Incorrect definition = incorrect interpretation. I edited this a few times but the separation is you can use an abstraction even if you maintain access to the implementation details.
> assembler
Assembly language which is a different thing. Initially there was no assembler, someone had to write one. In the beginning every line of code had direct access to all memory in part because limited access required extra engineering.
Though even machine code itself is an abstraction across a great number of implementation details.
> I don't see this as an abstraction as much as a simple engineering compromise (of accuracy) dictated by constraint (CPU time/solenoid wear/whatever), because you're not hiding complexity as much as ignoring it.
If it makes you feel better consider the same situation with 5 senators X of which have failed. The point is you don’t need to consider all information at every stage of a process. Instead of all the underlying details you can write code that asks do we have enough information to get a sufficiently accurate speed? What is it?
It doesn’t matter if the code could still look at the raw sensor data, you the programmer prefer the abstraction so it persists even without anything beyond yourself enforcing it.
IE: “hiding details that otherwise might make access more challenging”
You can use TCP/IP or anything else as an abstraction even if you maintain access to the lower level implementation details.
> You are thinking of assembly language which is a different thing. Initially there was no assembler, someone had to write one.
This is why I specifically mention opcodes. I've actually written assemblers! And...there's not much to them. It's mostly just replacing the names given to the opcodes in the datasheet back to the opcodes, with a few human niceties. ;)
> consider the same situation with 5 senators X of which have failed
Ohhhhhhhh, ok. I kind of see. Unfortunately, I don't see the difference between abstraction and encapsulation here. I see the abstraction as being speed as being the encapsulation of a set of sensors, ignoring irrelevant values.
I feel like I'm almost there. I may have edited my previous comment after you replied. My "no procrastination" setting kicked in, and I couldn't see.
I don't see how "The former is about semantic levels, the later about information hiding." are different. In my mind, semantic levels exist as compression and encapsulation of information. If you're saying encapsulation means "black box" then that could make sense to me, but "inaccessible" isn't part of the definition, just "containment".
Under the assumption that the input data is properly formatted you can generate machine code. This is however an abstraction which can fail as nothing forces a user to input valid files.
So we have an abstraction without any encapsulation.
But, that's a great example! Thank you. This makes it clear that I'm probably sometimes wrong. ;)
A really simple abstraction in mathematics is that of numeric basis (e.g. base 10) for representing numbers. Being able to use the symbol 3 is much more useful than needing to write III. Of course, numbers themselves are an abstraction- perhaps you and I can reason about 3 and 7 and 10,000 in a vacuum, but young children or people who have never been exposed to numbers without units struggle to understand. Seven… what? Dogs? Bottles? Days? Numbers are an abstraction, and Arabic digits are a particular abstraction on top of that.
Without that abstraction, we would have insufficient tools to do more complex things such as, say, subtract 1 from 1,000,000,000. This is a problem that most 12 year olds can solve, but the greatest mathematicians of the Roman empire could not, because they did not have the right abstractions.
So if there are abstractions that enable us to solve problems that were formerly impossible, this means there is something more going on than “hiding information”. In fact, this is what Dijkstra (a mathematician by training) meant when he said:
The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise
When I use open(2), it’s because I’m operating at the semantic level of files. It’s not sensible to think of a “file” at a lower level: would it be on disk? In memory? What about socket files? But a “file” isn’t a real thing, it’s an abstraction created by the OS. We can operate on files, these made up things, and we can compose operations together in complex, useful ways. The idea of a file opens new possibilities for things we can do with computers.
I hope that explanation helps!
To continue with the idea of numbers, let’s say you asked someone to add 3 and 5. Is that encapsulation? What information are you hiding? You are not asking them to add coins or meters or reindeer. 3 and 5 are values independent of any underlying information. The numbers aren’t encapsulating anything.
Encapsulation is different. When you operate a motor vehicle, you concern yourself with the controls presented. This allows you, as the operator, to only need a tiny amount of knowledge to interact with an incredibly complex machine. This details have been encapsulated. There may be particular abstraction present, such as the notion of steering, acceleration, and breaking, but the way you interact with these will differ from vehicle to vehicle. Additionally, encapsulation is not concerned with the idea of steering, it is concerned with how to present steering in this specific case.
The two ideas are connected because using an abstraction in software often involves encapsulation. But they should not be conflated, out the likely result is bad abstractions and unwieldy encapsulation.
To me, abstraction is an encapsulation of some concept. I can't understand how they're practically different, unless you encapsulate true nonsense, without purpose or resulting meaning, which I can't think of an example of, since humans tend to categorize/name everything. I'm dumb.
In sciences and humanities abstraction is applied the proper way, studying the instances first then describing multitude of existing phenomena by giving names to their common repeating descriptions.
The code always ends up nice and clean and modular. And, since I'm working towards the idea, I can say "here are the modular pieces I have to work with, dictated the fundamentals beneath, how do I use them to accomplish the task?". When working from the idea, I think it's easier to want to write something to achieve the immediate task, in a "not see the forest for the trees" kind of way (abstractions are about the goal, rather than the reality under). Of course, both directions are required, but I get the best "separation of concerns" going in reverse.
No code runs as fast as no code.
I can sort of understand it if I squint: every feature is a maintenance burden, and a risk of looking bad in front of users when you break or remove it, even if those users didn't use this feature. It's really a burden to be avoided when the point of your product is to grow its user base, not to actually be useful. Which explains why even Fischer-Price toys look more feature-ful and ergonomic than most new software products.
Amount of code is a huge factor but maybe not the best wording here. It's more a thing of complexity where amount of code is a contributing metric but not the only one. You can very easily have a feature implemented in a too complex way and with too much code (esp. if an LLM generated the code but also with human developers). Also not every feature is equal.
> Occasionally writing more verbose code is preferable if it makes it easier to understand.
Think this is more a classic case of "if the metric becomes a goal it ceases to be a metric" than it being a bad metric per se.
“My point today is that, if we wish to count lines of code, we should not regard them as ‘lines produced’ but as ‘lines spent’: the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger.”
Get two other, different, LLMs to thoroughly review the code. If you don’t have an automated way to do all of this, you will struggle and eventually put yourself out of a job.
If you do use this approach, you will get code that is better than what most software devs put out. And that gives you a good base to work with if you need to add polish to it.
To me, the solution isn't "more AI", it's "how do I use AI in a way that doesn't screw me over a few weeks/months down the line", and for me that's by making sure I understand the code it generated and trim out the things that are bad/excessive. If it's generating things I don't understand, then I need to understand them, because I have to debug it at some point.
Also, in this case it was just some unit tests, so who cares, but if this was a service that was publicly exposed on the web? I would definitely want to make sure I had a human in the loop for anything security related, and I would ABSOLUTELY want to make sure I understood it if it were handling user data.
We can have a discussion without the stakes being so high.
And we'll just have the AI write the postmortem, so no big deal there. ;)
SHOW AN EXAMPLE OF YOU ACTUALLY DOING WHAT YOU SAY!
It’s not perfect so I still review the code myself, but it helps decrease the number of defects I have to then have the AI correct.
by level of compute spend, it might look like:
- ask an LLM in the same query/thread to write code AND tests (not good)
- ask the LLM in different threads (meh)
- ask the LLM in a separate thread to critique said tests (too brittle, testing guidelines, testing implementation and not out behavior, etc). fix those. (decent)
- ask the LLM to spawn multiple agents to review the code and tests. Fix those. Spawn agents to critique again. Fix again.
- Do the same as above, but spawn agents from different families (so Claude calls Gemini and Codex).
—-
these are usually set up as /slash commands like /tests or /review so you aren’t doing this manually. since this can take some time, people might work on multiple features at once.
In addition to that, we do use CodeRabbit plugin on GitHub to perform a 4th code review. And we tell all of our agents to not get into gold-plating mode.
You can choose not to use modern tools like these to write software. You can also choose to write software in binary.
If you do use this approach, you'll find that it will descend into a recursive madness. Due to the way these models are trained, they are never going to look at the output of two other models and go "Yeah, this is fine as it is; don't change a thing".
Before you know it you're going to have change amplification, where a tiny change by one model triggers other models (or even itself) to make other changes, which triggers further changes, etc ad nauseum.
The easy part is getting the models to spit out working code. The hard part is getting it to stop.
The equivalent of "draw me a dog" -> not a masterpiece!? who would have thought? You need to come up with a testing methodology, write it down, and then ask the model to go through it. It likes to make assumptions on unspecified things, so you got to be careful.
More fundamentally I think testing is becoming the core component we need to think about. We should not vibe-check AI code, we should code-check it. Of course it will write the actual test code, but your main priority is to think about "how do I test this?"
You can only know the value of a code up to the level of its testing. You can't commit your eyes into the repo, so don't do "LGTM" vibe-testing of AI code, it's walking a motorcycle.
FWIW we do have our docs on testing approaches here, and have recommended a more integrated-style approach to testing for a while:
I think the main issue I was having with it was reusing the store object instead of creating a new one for each test. The other issue I was seeing was that it was creating mock objects and API's for things that weren't even being tested (lot of scope creep), and one of those API's was basically copy-pasted between two files (code duplication). It was also just testing things that weren't really necessary (ie, testing Redux itself, instead of my usage of it).
Another issue was just taking a complex approach to fixing something that could be more easily solved. For instance, I had debug: true turned on for redux-undo so I was seeing some unnecessary log messages in the test. Codex identified this, and asked if I wanted to disable them, so I said yes. What it did though, was instead of setting debug: false, or disabling it on tests, it patched console.log to look for redux-undo prefixes. Technically worked, but kind of byzantine!
None of this was a terrible disaster or anything, especially since I started pretty small, but I think what made me miss some of the issues at first glance is this is my first usage of Redux in a non-toy project so while I understand the fundamentals fine, it was easy to sneak working-but-bad stuff past me until I sat down with the code to write a test on my own and started to see the issues.
This has been my experience almost every time I use AI: superficially it seems fine, once I go to extend the code I realize it's a disaster and I have to clean it up.
The problem with "code is cheap" is that, it's not. GENERATING code is now cheap (while the LLMs are subsidized by endless VC dollars, anyway), but the cost of owning that code is not. Every line of code is a liability, and generating thousands of lines a day is like running up a few thousand dollars of debt on a credit card thinking you're getting free stuff and then being surprised when it gets declined.