Bootstrapper909

I run claude code unattended on a linux box most of the day. the budget cap is nice, the harder part for me was telling "done" apart from "stopped".

Knowing whether it actually works. Writing the code got cheap, checking it against the real system didn't.

tmux kind of is that daemon. The agent keeps running on the box and I attach from my laptop or phone. What I was missing was everything around it, like paste and files.

We split them. The agent writing the code never writes the checks it's judged by, and those separate checks caught way more than its own tests did.

Very neat! Generally speaking, it seems like testing and debugging, hasn't been getting as much love as other aspects of computer programmeing.

Debuggers still work pretty much the same as gdb, only now with an integrated UI, and there's so much room for improvement. This is a great start and can be taken much further with abilities to mimic complete sessions to easily reproduce bugs and run them again and again until fixed.

We see the same at the related testing automation field (disclaimer - checksum.ai founder). Same testing methods, same testing problems, only fancier packages.

Very cool! Any specific reason you started with Java?

We have a few privacy controls in place:

1. We hash all inner text and then backfill static strings on the server side. So every text that is specific to the user remains hashed

2. We detect special cases like passwords, SSNs, credit cards, and completely block it (even not hashed)

3. We provide full privacy controls to our customers to easily mask any sensitive elements

4. We discard the user IP and don't require any PII to be sent. So we can connect a session together, but don't really know who the user is

Yep totally understand. We are an early stage startup and currently 100% focused on improving our models and our product.

We don’t have pricing, not because we try to be vague, but because we haven’t fully figured out our training costs, which can vary significantly per app. We are very much in the “Do things that don’t scale” phase where we hand-pick our customers, provide white-glove treatment and prioritize learnings over price

We train the model based on user sessions to learn how to use an app. The model learns how to execute specific flows, but also how to interact with components in the more general sense. Since most developers use compostable components, patterns of usage are repeated across the same app.

Then, during test generation, we bias the model to explore edge cases (in a few ways), and the model is still able to complete those even with low sample.

In other words, we direct the model toward certain goals, and flows and also add chaos to the process which result in the model executing unexpected flows.

Thanks for your feedback. It definitely makes sense and we'll incorporate it!

Thanks for your kind words! Yes many teams struggle with that (and I have in the past) and the essence of ur mission is to allow dev teams to focus on progressing on their roadmap and goals instead of wrestling with tests.

Feel free to sign up for a demo if that's a priority for your team. Even if it's just to chat and connect.

It's all of the above but more specifically:

1. We use AI to analyze the user patterns and find common paths and edge cases, basically building a representation of your UX in a DB

2. We then use the DB to train another ML model that learns how to use your app the same way a user does. Given a certain page and user context, the ML can complete UX flows.

3. Finally, we learn to generate assertions, run the tests and convert the model actions in step 2 into proper Playwright or Cypress tests

That's a fair comment and I guess we are missing and "AND" there.

1. We (and others) have tested our tools' impact on memory, CPU, network performance and found only negligible impact, even on slower/older devices

2. Also, they are used by F500 companies and have wide adoption, which indicates that other well established devs have run the same tests and decided to move forward.

We'll work on the language there to clarify.

We're currently focusing on web apps.

There's nothing "specific" in the underlying model that prevents it from testing mobile. It's just a matter of focus at the current time.

Our landing page at checksum.ai has a video in the hero section of test. We added some graphics (e.g. the green checkmark), but the steps executed are real tests that we generated.

But the tl;dr is 1. We learn how to use your app based on real sessions (we remove sensitive information on the client side) 2. We train a model on this data 3. We connect this model to a browser and generate Playwright or Cypress tests

The end result is code written and Playwright or Cypress. You can edit and run the tests regularly

I agree! My experience with test generation tools was also lukewarm which is why we founded Checksum.

> How is the product different from the other test generation tools

We train our models based on real user sessions. So our tests are: 1. Completely auto-generated 2. Achieve high coverage of real user flows, including detecting edge cases 3. Automatically maintained and execute with our models so they are less flakey.

> How do you check if the are testing the intended behavior

Our models are trained on many real sessions so it learns how your website (and others) should behave. In that sense, it's similar to a manual QA tester which can detect bugs. To supplement for functionality that is not obvious by the UI, we are now looking at adding LLMs to parse code, but most of the functionality can be inferred from the UI

That's a great question!

We actually use real user sessions to train our model, so when I use the term coverage our main metric is covering as many user behaviors as possible.

We collect data in a privacy-focused way essentially anonymizing all sensitive information, as we don't need to know the user specific context. Only the main flow.

I'm not sure it's just about constrains. There are some problems that are just too "random" for FP thinking. Usually those around human behaviour.

When planning your next startup, even with unlimited budget, FP might not get you far. You simply cannot predict product/market fit and how you should develop your product without user feedback

The question before can should be why?

Code being 2d and not immersive, I don't see any benefit using VR over a monitor.

Another comment: The article mention big "sign up" banners as a dark patterns. I don't think it is.

a dark pattern is either (i) getting the user to do something they don't intent to do, or (ii) preventing the user from doing something.

A big sign up banner might be poor/aggressive design, but as long as users choose to click on it, it's not a dark pattern