khasan222

I used playwright aria snapshots mainly, and less on screenshots because there is a lot of unnecessary info in the screenshot, but the aria snapshots do really well. Have you tried using aria snapshots for exploration too?

Also wow I have been building something similar to that 2 year endeavor. But I didn’t think to store the tests in the actual codebase because the idea was to never look at the codebase for the vibecoders and such. But reading all of this I’m rethinking my technique there, why have the artificial constraint?

Why not use ai to update the tests first? Make sure the tests conform to the old codebase, then use those same tests to verify the new functionality?

I feel you, and I think we agree, I mean it depends with the intern writing code, because I always ask myself, if this code were illegal who would go to jail to determine ownership, and shucks, I never considered the intern angle :). I don't think I would be culpable there, but i would definitely be at fault if it were authored by ai.

and about my tacit love for software you're right I don't want to write code just for the sake of it, for sure, I want to write code towards a purpose, always with some goal in mind. But I gander, we are all like that, because how much time do we spend planning and deciding before ever putting fingers to the keyboard.

But it does suck sometimes, I'm just trying to find the joy in the new parts, ya know? I want to embrace the new technology, and use it, because the toothpaste is out of the tube, and it can't go back in. Everything else I do or believe stems from that belief.

lolol :) man who knows, i just be rambling.

I have found, that the best way to prevent bad output with AI is proper planning, and also extensive end to end tests. - If it is implementing a design, I have it compare the screenshot with the mockup and get 97% similarity. - If it is a functionality, I have it spin up a browser/tv/mobile and check the output is what i asked for.

I also use plannotator for planning, that is an amazing tool! need to buy them some coffees for sure.

I want to agree with you, and my instinct certainly is to do so. one thing is for me is, I don't see code as art, to me it is math / science, and we are scientists, using the scientific method to solve problems with questions, hypothesis, evidence, and conclusions, all in the form of documentation and code.

I love writing code, it is quite thrilling going through meticulously and spotting that thing no one else could. Writing code, however, is daunting, taxing on the mind, takes a lot of mental toll, and you must be in the proper mental space. So if i have a tool, that can take some of that weight off of my mind, so i can focus on the bigger picture, is a good thing.

The end goal is still software that people use. I think we should focus more on our new found extent of our powers. I feel at least 6x my normal velocity. How many new projects have you been able to tackle since the advent of AI? I know for me it has been numerous for sure. We're living in the future man. I write code on the train, and in my sleep nowadays. Crazy times.

One thing I feel like a lot of us are missing is that we no longer have to worry about looking for a missing semicolon, typo, or off by 1 error, or manually combing logs to try and understand a problem.

Sure we have to do a lot more work, but i would say it is less mundane than it was. Now i get to focus on the fun parts of software engineering. I think the burnout some of us are feeling is because as the author has said we don't know how to properly use the tool.

Those techniques will come with time through us using the tools, and coming up with the standard for how to use it for the most productivity gains. For instance how many of us, when manually coding, have to have headphones with noise cancellation, or use the pomodoro technique to get things done. I imagine soon we will have similar techniques for working properly with AI.

I think people will get way better at wrangling code with ai as time passes

Isn’t it saas that facilitates a network with a specific goal in mind?

Idk if I disagree completely but it feels like I do. To elaborate, I don’t think we will see less applications, I think we will see more with better features.

My reasoning is people aren’t happy with many of the titans we have for certain apps, uber is my favorite example. Everyone hates the price it charges thinks it’s unfair, but to go back to a time before uber is unfathomable.

That to me is a ripe market to be overtaken. What it really needs is a taxi driver or operator so fed up with the current system they spend their energy organizing and making their own uber with less greed.

For sure not every tool will become huge, but most of the tools we use nowadays started with someone having a problem, being frustrated with the status quo, and having the will and means to execute.

The means necessary to execute get lower and lower everyday.

What do you mean when you say public interfaces?

I am still skeptical of ai running everything, given how much it changes and quickly, I’ve not quite given over all control. And I try and use it to learn more about things normally I wouldn’t have time for.

The goal of writing the git messages is to slow down and prevent a worst case scenario where work is lost. Given how powerful git is I think giving control over is like doing a sure fire migration without a backup in the way that it can lead to easily preventable problems that are difficult to fix afterwards, it’s just a bad paradigm.

I can appreciate some parts of this, like keeping a painfully detailed record of the changes written by ai, I have this too, but it is separate for the content meant for actual human eyes.

Like others here I find a hard time finding specific evidence or reasons for some of op’s thoughts, but in general it just seems like a recipe for problems when you’re too trusting with ai with all the processes

Ngl I’m doing this right now for a client. Part of my strategy is to write out e2e tests that get a certain baseline of functionality, and then use that as the check for any change that I make to the codebase to make sure it continues to work.

So workflow for a full web app is make e2e tests for all use cases. Then add a very strict duplication checker, and linter, and then just tell the ai to hit a certain duplication limit like 3%, check the linter, and add unit tests to ~95% or greater of the code.

With the right CI and other checks that are deterministic you can really do a lot with a codebase.

My last sentence is definitely aspirational, it is how I try and go about it, but for sure I make mistakes. However your comments about writing spec tests was interesting to me.

Honestly I don’t even write tests manually because of coverage checks. Being that the coverage check is not something easily manipulated, I always tell the ai, don’t ever change configs, and make the coverage pass whatever I set it to, most times > 95%. I just tell the AI, make this coverage pass.

I find tremendous success with this technique, or anytime really I can find an objective way for the ai to test its work.

For sure every time you use ai you’re sacrificing understanding if you don’t plan out and understand how exactly the ai is going to do the work you asked it to do.

The same output that is such a bad thing in this article can also be used to gain context, by making a thorough plan with your ai first, reading through the plan and proposing changes just like you would with a real developer.

You can also use this output to have the ai write a journal as well. The journal can be as detailed as possible and essentially a ledger of all of the changes your ai has made to the code. This allows not only for your teammates reviewing your pr to gain greater context, but also can be used by yourself, or even the ai itself to figure the why behind a particular implementation was done the way it was, far into the future even.

Lastly how many of us ever deploy code without actually checking the feature works e2e? I would gather not many of us do, I don’t, because even though we may have a greater understanding of the code, we can make mistakes in the code or in our logic. And I keep coming back to why would we treat llms any differently? I believe we should be spending our energy thoroughly manually testing a feature to make sure when we brainstormed we actually did get every edge case, and it works well.

> I always wondered why people don't also ask the AI to generate code comments/documentation, summaries of those documentation, overview of the system, and re-review them all for correctness for the changes they asked the AI to do.

I now on all of my projects have an ai journal that stands as a ledger for every change the ai has made, and why it was made. I don’t read it that hard personally because I spend so much time planning with my agent before letting it code. However I have found it very useful in sharing code between people, or having Claude look through the journal to gain context when modifying or adding a feature.

I’m not very familiar with Go, however after looking at the repo I can’t help but notice there is no infra to ensure code quality. Do others see the same thing, because if so that is the real problem

Yes I agree for sure llms write terrible code when left to their own devices, but so do most engineers. Which is why we have so many tools to help keep a certain level of quality. Duplication checks, tests, linters, other engineers.

I find whenever you make an llm repo without these checks, and more, it will write like an enthusiastic junior engineer, wrong and strong. However a junior engineer would be hard pressed to get 95% coverage on a codebase, the ai is more than willing and does it in a few minutes. We can use things like this to our advantage, how many people have ever seen a repo with 100% test coverage? With ai this is very possible, with people not so much.

LLM’s writes terrible code, we know this, but when dealing with humans that write terrible code we have many techniques. Let’s use those same techniques to keep the llms honest, but more importantly verifiable.

Weirdly, and i fully think this is just some cognitive bias I don't have the knowledge to name, the ai seems very happy to please me. Like when it gets something done in one shot, it seems very happy to do so.

Well if the spec is incomplete it sounds like you should lower scope for the AI, and then go from there. I wouldn't be too keen to give a junior engineer free reign and expect awesomeness

Completely agree! People tend to forget we are non deterministic too! Yet we are able to write code fine, and fairly reliably by using tools that can help keep us fairly honest.

I think most problems with ai tend to be around can you deterministically test the thing you are asking it to do?

How many of us would never ever show work, without going to check the thing we just built first?

Yeah but that is how many tools just to get it to work, and how much burden on your PC. it just seems simpler to me to just use as few tools as possible to accomplish the goal.

Also shameless plug, I wrote something about this very thing. https://khalah.medium.com/getting-ai-to-work-right-27b750dba...

I actually now think ai prompt writing in the IDE is completely overkill nowadays.

IDEs are made for just a human to interact with code. I think the paradigm of forcing these tools that weren’t built for this to do this, is us trying to fit a square peg in a round hole.

Call me old, but don’t put ai in my ide. My ide was made for a human, not an ai. For the established players for sure it makes sense since they already have space on our machines. But for the new ones imo terminal, or dedicated llm interfaces are where it’s at.

If I’m writing code sure suggest the next line. If the machine is writing code, let it, and just supervise properly. and have the proper interface that allows the strength of each

I was just thinking the other day how ai will be way worse than social media in terms of influencing people. Before though I was thinking of times people have been convinced by chat bots they were god or a genius.

Social media was so powerful because it convinced people their world view was popular and correct even if it definitely wasn’t, but at least then it was actually happening somewhere. This is just completely made up, to feed into people’s world view, and they don’t even have to find the perfect figure they can make one up.

We’re definitely going to suffer a lot before it gets better. Interesting times we’re living in.

Oh yeah the natural images are also low quality. Email me I’ll share with you something else I built

Thank you for the feedback. The store isn’t done waiting on the friend for actual products with their images etc.

If you really want to see what I was messing with email me. I’ll share on non public forums.

Eh, for many reasons I am not posting it here. It is a passion project for something and would lead to problems if I post it here. That being said I was trying to share the technique.

The reason for the post is that even without the actual website one should be able to envision the technique and how it may or may not work. Also if you look above recently I added links to the Claude.md for another thing I was working on for a friend that also had to solve this problem.

Just want to give people the tools to use ai well from my own findings

Thank you. Yes took a bit but still way faster than by hand. There are other store pages that are also implemented. This 1 page took me like an hour lol.

Hey, one thing I made with this technique is hartwork.life a simple Wordpress store for a friend. I used open ai to design it for me, and then used the techniques above to get Claude code to implement the proper designs.

I am still trying to learn how to wrangle Claude properly, but I have this Claude.md[1] for that I used to make the website. In particular one of the last rules about using imageMagick for comparison.

I haven’t touched this website in a bit (waiting on client) so now I use playwright mcp for the screenshots and the browser interactions.

[1] https://github.com/panda01/hartwork-woocommerce-wp-theme/blo...

I built a few websites, most of them it wouldn’t be wise to place on here. But someone emailed me about this, so I’ll do my best to help I did build https://hartwork.life for a friend with a design from open ai (pre google stitch which is my current preferred tool)

Here is the line from my Claude code to get something like this. Keep in mind I didnt use mcp for playwright with this particular implementation but it is my preferred method currently. Tha

CRITICAL - When implementing a feature based off of an image mockup, use google chrome from the applications folder set the browser dimensions to the width and height of the mockup, capture a screenshot, and compare that screenshot directly to the mockup with imagemagick. If the image is less than 90% similar go back and try and modify the code so that way the website matches the mockup closer. If a change you make makes the similarity go down, undo it, and try something else. be mindful the fonts will never be laid out exactly like the mockup, please use blur at a max of 10% to see if the images are closer matching. If you spend more than 10 cycles screen-shotting and comparing, stop and show the user how similar they are mentioning any problems

The more text the harder it becomes and it’s why we really need the blue because fonts are almost always rendered differently.