0x63_Problems

Active work should mostly be hillclimbing, or at least more exploratory. The search space is very large and you typically find a lot more low-hanging friction than you'd think by just running a broad battery of sessions. And I'd pair active hillclimbing work with feature launches/product work - essentially part of releasing new things is QA-ing them against coding agent workflows.

The test suite eval style is useful and mostly a complement. You do want regression tests on critical features.

To clarify, this does not require the hillclimbing to be unattended, especially if you are setting up the system for the first time!

This mainly automates the process that you are doing by hand. If you have your own definition of friction and priorities, you can tell this to your agent and it will fit into the CLI loop. You can review any change that is suggested, and spot-check transcripts if you feel the need.

It's like most AI systems in that you can have it earn more autonomy over time, as you add guidance and guardrails.

Fair enough, maybe when we use it internally it would qualify?

You can reproduce that finding directly from python:

```

>>> request.urlopen("https://anc.dev")

[...traceback info...]

urllib.error.HTTPError: HTTP Error 403: Forbidden

```

In this case (https://agents.withgauge.com/p/runs/40be763c-4678-4677-88df-...) the agent wrote a python script to test the site, and hit the error above. We don't have a full filesystem diff viewer in ax-check.com, although we do have that in the Gauge agents product itself.

which URL is it? it's kind of a deep assumption since we're evaluating on ease-of-navigation from the homepage, and looking for /llms.txt etc. but there should be a way for me to add support

Behind an agent is a human that might use the product. I think customers/consumers will demand that their personal/subscription agent can access the product.

Each product is different (how much does it cost to serve a marginal user, how risky is it). But generally I think allowing ephemeral accounts like Cloudflare, or serving simulated/static accounts before a human claim can both work.

For a lot of products it can just be an extension of the free tier I think too.

Are you mainly concerned about scraping? or anti-agents in general?

The initial step of the scan looks at homepage content to try to figure out where to navigate next, it's getting confused by all the different products linked. Will fix this!

I think it's only recursive from the perspective of the humans, i.e. they design Astra, which itself as part of its deployment designs Galactica, etc.

So humans develop things one after the other, but when the thing itself starts developing new things, those are happening 'recursively' in its scope.

Gauge | Founding Engineer | San Francisco, CA | ONSITE | Full-time

Gauge helps companies show up in more AI answers. Search traffic is shifting away from Google into AI chat, and companies are figuring out how to adapt right now. Our system delivers real-time analytics derived from millions of LLM answers, while providing an intuitive, powerful UX. We also generate optimized articles to improve our customers’ visibility, and offer an agentic chat interface over their data (think ‘Why was there a spike on 12/10?’ or ‘Which of my pages should I refresh?’).

I'm Evan, the CTO, and we're looking for engineers with strong programming fundamentals, creative problem solving skills, and a high-agency working style. You might be a good fit if you are an ex-founder, ex-founding engineer, or have worked at a Seed/Series A startup.

We primarily use Django, React, Postgres, Celery, and Clickhouse.

Contact me with your resume (and mention HN) if you're interested: evan[at]withgauge.com

More details/JD also here: https://www.ycombinator.com/companies/gauge/jobs/qemTjv7-fou...

Very cool!

Yup I definitely see confusion in our responses around the product and brand names. We do another pass through an LLM specifically aimed at ‘canonicalizing’ the names, but we’ll need to get more sophisticated to catch most issues.

In that case you mentioned, the brand confusion is what accounts for the top three omission for QBO. Both OpenAI and Perplexity rank it #1, but Anthropic ranks the slightly different “Quickbooks” product as #1. Our overall ranking prioritizes products that appear in all three responses, so both are dropped down.

No specific reason, just started with these three, will add Gemini soon!

It's from previous searches actually, we have an 'enrichment' step after the initial rankings come back which helps with semantic deduplication and tries to give us a canonical website domain. We store the Product and tag all matching rankings: https://productrank.ai/product/microsoft and use a 3rd party to map website <-> brand logo.

We are using sonar-pro

These are structured results from explicitly asking the LLM for a ranking in the given category, and we provide guidance in the prompt telling the LLM to 'use best judgment' when the topic doesn't clearly include products.

Also we include the 'key features' from each answer - you can see this by clicking the cell containing the rank (e.g. '1st' in the Anthropic column)

In this case, Anthropic said of 'Death during sleep':

Anthropic Analysis for Death During Sleep

    Painless and unaware experience
    No anticipatory anxiety
    Common with certain cardiac conditions
    Often described as 'peaceful'
    No suffering

I actually made that change in response to their comment! It used to be a live GIF with no narration.

Yes!

Here is an example project that is configured as if it were a uv workspace: https://github.com/gauge-sh/tach/tree/main/python/tests/exam...

In that project, `tach check-external` would handle between-workspace dependencies, while the core `modules` and `interfaces` config would handle within-workspace dependencies.

Soon these will be better unified, we kept the 1st-party/3rd-party distinction separate while we learned what the UX should be.

This should definitely be supported out-of-the-box, we can take a closer look in Discord or through GH Issues! But generally I would expect:

```

[[modules]]

path = "foo"

depends_on = []

[[modules]]

path = "bar"

depends_on = ["foo"]

```

would do the trick, assuming both are within a configured source root, and their children are _not_ also marked as modules. If the children are marked as modules, their dependency rules are treated separately and wouldn't automatically inherit from a parent.

I can understand the frustration, but I think there are legitimate reasons to run this remotely.

Tach is an installable Python package, shipping a full web app would have to come in a separate form factor and has significant maintenance implications. Given we are explicit about the remote app before anything is sent, require explicit opt-in, and we provide usable alternatives locally, we prioritize shipping a useful graph experience that is immediately usable.

If you are at an enterprise that cannot tolerate this, then you can use a local viewer with either GraphViz DOT format or Mermaid which is generated by using `tach show` or `tach show --mermaid` respectively.

Co-author here - We do provide a web platform (https://www.gauge.sh/platform) which we have been developing with design partners. The fundamental difference between using Tach alone vs. the platform is that the platform provides incremental enforcement at the pull request level.

We're always happy to chat about adding more design partners! email: [email protected]

Co-author here, I can understand where you're coming from!

Part of the philosophy here is that the tools and techniques you're describing can (and should) be used diligently to solve this problem, and Tach is often a complement to this approach.

The benefit of centralizing the concern into a single tool, and often a single config file, is that teams get better documentation, earlier feedback (in-editor vs. code review), and more visibility when planning new development. Teams also get to choose _how_ they would like to satisfy Tach's config, and other teams can still rely on the same guarantees due to Tach's static checks.

Co-author here, fair question!

In short, we want to make the visualization UX as smooth as possible, and this is best done with a web app. The URLs use UUIDs, and the contents being sent don't include literal source code, only module names and Tach configuration. We will also delete graphs by UUID on request, and have done so in the past.

That said, we do try to be up-front about this, which is why that disclaimer exists, and when running this command on the CLI, you must supply an explicit `--web` argument to `tach show`. Otherwise, the default behavior is to generate a GraphViz DOT file locally.