ronpotok

There are 3 mechanisms for context changes / staleness - one automated, one reactive, and one silent.

Automated: Through github actions, you can hook your data processing repo (like dbt) to this repo, so that when a change occurs upstream, it reruns the evaluations like a regression suite. If there are failures, you can add your AI API key (currently only Anthropic, but on the list to expand) and it will suggest changes to the context.

The reactive mechanism - where the business starts asking questions they never have before. I am building observability tools (seen nodaldata.io) specifically around that process. This is where new context typically needs to be added.

The third way context goes stale is that the business quietly changes a definition - “active customer” now means something different, without a dbt change or a "new question" being asked. Including dashboard reconciliation in occasional regression tests could surface drift when a dashboard trusted number and the eval number diverge.

Also, since every definition carries an approver name and a confirmed date, there may be an "interview refresh process" (maybe every 6 months) where analysts re-interview for stale documentation.

In addition to the observability tools that highlight new questions, I try to instill sampling into self service analytics to continually monitor performance of the system.

Have you seen any other techniques that could work?

The typical user is a data analyst (who first have been given permissions to the query history by the data admin).

The AI performs an interview with that analyst using their AI of choice, then the analyst can hand the interview off, through Github, for another analyst to build a second domain's context (if there are multiple analysts).

Making phoneme models for children is difficult (or at least was a couple of years ago). Most speech recognition models are word recognition models, not sound recognition. I built some of the best ones a couple of years ago in my company, Kid Smart AI. Unfortunately I sold that company last year to a large edtech. Hopefully someone else will pick up this use case soon.