drothlis

I guess you mean its "Computer use" API that can (if I understand correctly) send mouse click at specific coordinates?

I got excited thinking Claude can finally do accurate object detection, but alas no. Here's its output:

> Looking at the image directly, the SPACE key appears near the bottom left of the keyboard interface, but I cannot determine its exact pixel coordinates just by looking at the image. I can see it's positioned below the letter grid and appears wider than the regular letter keys, but I apologize - I cannot reliably extract specific pixel coordinates from just viewing the screenshot.

This is 3.5 Sonnet (their most current model).

And they explicitly call out spatial reasoning as a limitation:

> Claude’s spatial reasoning abilities are limited. It may struggle with tasks requiring precise localization or layouts, like reading an analog clock face or describing exact positions of chess pieces.

--https://docs.anthropic.com/en/docs/build-with-claude/vision#...

Since 2022 I occasionally dip in and test this use-case with the latest models but haven't seen much progress on the spatial reasoning. The multi-modality has been a neat addition though.

Does your model consistently get the positions right? (above, below, etc). Every time I play with ChatGPT, even GPT-4o, it can't do basic spatial reasoning. For example, here's a typical output (emphasis mine):

> If YouTube is to the upper *left* of ESPN, press "Up" once, then *"Right"* to move the focus.

(I test TV apps where the input is a remote control, rather than tapping directly on the UI elements.)

aka snapshot tests.

According to the GLIP paper,† accuracy on a test-set not seen during training is around 60% so... neat demos but whether it'll be reliable enough depends on your application.

† https://arxiv.org/abs/2206.05836

That would allow a lot of caching optimisations, as you can "index" each file in isolation.

[1]: https://github.blog/2021-12-09-introducing-stack-graphs/

[2]: https://github.com/github/stack-graphs

- Snapshot testing is like version-control but for the outputs rather than the inputs (source code).

- Asserts in traditional unit tests are like "block lists" specifying which changes aren't allowed. Instead, snapshot testing allows you to specify an "allow list" of acceptable differences (e.g. timestamps).

I'm not particularly wedded to any of these terms, I'm just pointing out that "regression testing" has an established meaning, and it isn't snapshot testing (outside of certain industries, at least). I do find it amusing that one implementation of snapshot testing (https://pypi.org/project/pytest-regtest/) links to https://en.wikipedia.org/wiki/Regression_testing but that article doesn't describe snapshot testing at all! Maybe the article changed? Oh well, language changes too. ¯\_(ツ)_/¯

"Regression test" means something else, at least at the companies I've worked at: It means a test that was written after a defect was found in production, to ensure that the same defect doesn't happen again (that the fix doesn't "regress"). It can be a manual test or an automated test. https://en.wikipedia.org/wiki/Regression_testing

I suppose we could use systemd-nspawn on our cloud servers too, instead of docker, but when we wrote the build system we were already using docker so it was the expedient thing to do at the time.

OSTree is still working very well for us. At the time I wrote the article we had been using OSTree (and the build system I described in the article) for 4 years; 3 years later not much has changed. In those 3 years we have started building images for different architectures and our build system / OSTree handled it just fine, as you'd expect (it already handled cross-compilation, but for a single architecture).

Our build system also builds Docker containers (for our cloud services) but instead of a Dockerfile we use a custom yaml format that lists the base image (like "FROM" in a Dockerfile), the apt packages to install, and the commands to run (like "RUN"). Then we create a lockfile of all the apt packages, download them (into a local OSTree repository, but that's an implementation detail), and install them with a custom "docker run" command + "docker commit". We end up with the base layer + the apt layer + a single layer produced by concatenating all the RUN commands + a layer with our compiled Rust binaries and Python files. We use apt2ostree to generate the lockfile (really it's our patched version of aptly doing the work) but we use docker (not OSTree) to build the Docker layers. We use Docker's standard push/pull mechanisms to deploy these containers.

To hook this up to Ninja we use "marker files" (a file in the build directory) to track whether this work has been done (e.g. you need to regenerate the apt layer if that layer's "marker file" is older than the lockfile).

Our Python "configure" script (i.e. our build system) looks something like this:

    def build_docker_container(name):
        yamlfile = f"{name}/docker-base-image.yaml"
        with copen(yamlfile) as f:
            data = yaml.safe_load(f)
        ubuntu_version = data["ubuntu_version"]
        image = docker_pull("ubuntu:{ubuntu_version}")
        image = docker_apt_install(image, data.get("apt_dependencies"),
                                   lockfile="%s/Packages-%s-amd64.lock" % (
                                        name, ubuntu_version),
                                   ubuntu_version=ubuntu_version)
        cmd = "..."  # RUN commands from yaml
        deps = [...]  # dependencies from files listed in yaml
        return docker_mod(image, cmd, deps)

What I really like about the Python+Ninja combo is that you can pass these targets around as Python variables — you don't have to come up with an explicit filename for each one (the target name is generated by each helper function, e.g. `docker_pull` returns "_build/docker/${name}"). This makes it so convenient to compose these build rules. And if you ever need to debug your build system, everything is very explicit in the generated ninja file.

We don't have a ton of these containers so it has worked well enough for us because the apt layer changes rarely, and the layer above it is small/fast. The main thing we get (that you don't get from vanilla docker/apt) is lockfiles and reproducibility.

- Server-side rendering and testability: https://david.rothlis.net/server-side-rendering-good/

- Testing "Page Objects" (classes written for UI tests): https://david.rothlis.net/pageobjects/characterisation-tests

In my (artificial) benchmark, make scaled poorly, taking 70 seconds to process 100k C files worth of dependencies, vs. ninja's 1.5 seconds: https://david.rothlis.net/ninja-benchmark/

Most of make's time was spent processing the ".d" files containing header dependencies (Ninja has a special optimisation for these files, where it reads them the first time they're created, inserts the dependency information into a binary database, then deletes them so it doesn't have to parse them in future invocations).

In real world projects, you often end up "abusing" make to add behaviour such as detecting if the compilation flags have changed, and this can make your makefiles slower; whereas ninja has those features built in. Apparently this made a big difference in build times for Chromium (where ninja was born). See this comment by the ninja's author: https://news.ycombinator.com/item?id=23182469

I reported that here: https://lists.gnu.org/archive/html/help-make/2016-11/msg0000...

...and the maintainer said that the behaviour isn't intentional, but after a short look at the source code I decided fixing it was going to be beyond my abilities/motivation.

(I suppose this is more relevant to full builds, not incremental ones.)