schmuhblaster

Maybe I missed in when scanning the paper, but did they compare to a simple self modifying harness, e.g. instructing Pi to update some code or skill docs based on the results?

Turning 100+ page SOP documents into executable policies using DeepClause and DML.

Building on the self-modifying harness concept we show here how to use DeepClause to build more reliable agents for outsourcing work defined via SOPs and policies. Instead of taking a big markdown file and putting it into the system prompt, we let a coding agent convert policies into small executable logic programs in a DSL called “DML”. The “leaves” of these programs can either be deterministic rules or LLM-driven agent loops. DML programs run safely inside the WASM build of SWI-Prolog.

Great work! If anyone is looking for a way to integrate something like this into their own harness or the pi coding agent, then you might be interested in DeepClause [0]. It comes with a Prolog-like language implemented on top of SWI-Prolog (WASM Version). The purpose of the project is to allow for broad experimentation around the intersection of LLMs/Agents and GOFAI. So you could use it to build memory systems like OP did, create executable specs, define graphs and loops for agents and subagents... It also comes with a pi extension that greatly simplifies getting started with it.

Opposed to OP, DeepClause uses Prolog semantics, so running some more complex queries on knowledgebases might cause some issues (which is the use case where a Datalog might be more useful). For smaller scales it should be fine though.

[0] https://github.com/deepclause/deepclause-sdk [1] https://github.com/deepclause/deepclause-pi

I've been super impressed by DuckDB ever since first trying it out about 3 three years ago. Congrats to the founders!

I think that writing your own harness is a rite of passage now, just like writing your own search engine or database, rolling your own crypto…

Anyways, please try mine!

https://github.com/deepclause/deepclause-sdk

That should be possible. You can e.g. take a look at the deep research example skill.

The Prolog/DML lets you express any kind of loop, graph, whatever workflow.

I’ve been working on DeepClause [0], an agentic harness, the core of which is based on Prolog. Every interaction (conversation turn, plan, execute plan, etc.) triggers a small logic program that orchestrates one or more agent loops or prompts. All these programs are completely hackable, so you can easily adjust the core logic and behavior of your agents in a reliable manner. Also, instead of using markdown specs, you can “compile” your markdown into Prolog code. My hope is that this might unlock new possibilities for spec driven development.

It’s been a lot of fun and I am somewhat proud of Prolog/Typescript integration layer built on SWI Prolog’s WASM version.

Other than that, I am not sure how and if I will continue with it. Feedback welcome!

[0] https://github.com/deepclause/deepclause-sdk

Indeed, and maybe that's all there is to it. Still, I'd hope we will eventually better understand what's exactly happening in the wake of many repeated applications of f().

Yes, I am the author and thanks so much for trying! Please do submit a github issue. My first suspicion about the speed is that maybe an inner loop is taking too many turns until the model finally realizes that a task is finished (so that in turn the runtime knows whether the predicate failed or not and can continue execution accordingly). Happy to take a closer look!

The point about multiline prompts is very valid obviously, that's on the todo list.

Sorry, did not notice your comment until just now.

So far I am observing two things:

1. For smaller models, performance on Benchmarks such as DeepPlanning does increase significantly.

2. Context hygiene for sub agents becomes much simpler, since that can be expressed relatively concise and the mechanics are handled by the runtime automatically.

Still looking for a good test cases to study possible advantages, but running reliable benchmarks does take time and money...

For my own (rather idiosyncratic) harness I've been experimenting [0] with "compiling" long markdown specifications into small executable logic programs. It's too early to tell for sure, but I believe that this approach does have its merits when you want some guarantees about how your agents behave for longer tasks.

[0] https://github.com/deepclause/deepclause-sdk

You might like this: https://github.com/deepclause/deepclause-sdk.

It’s a DSL I’ve been working on to encode mixed deterministic/probabilisitic agent behavior.

Been building various LLM+PDF pipelines at work. As soon as you need to e.g. parse tables etc. it becomes a lot of hard work!

As a European I have long given up on any meaningful change w.r.t AI. Imho the average European is much more risk averse than the average American or Chinese. That and a plethora of other factors that have been discussed over and over again, make it unlikely that we'll see things change within the next ten years or so. Only massive and immediate threats (e.g. he crisis in Ukraine) will make people and governments reconsider their fundamental beliefs (and even then the pace of change will be slow).

Awesome work! If I understand it correctly, it loads relevant subgraphs from the DB and then runs queries in Prolog on it? Or is it more similar to datalog?

It's my own (slightly idiosyncractic ;-) harness: https://github.com/deepclause/deepclause-sdk

I've worked extensively with the slightly less able cousin, the 35B A3B model and tuned my own harness around making it work well with local or non-sota models. The results are quite promising [0], if one sticks to a plan-execute approach. After a bit of fiddling with llama.cpp I was able to get it to work through a small change on a real codebase from work on a 32GB M5 (typical python FastAPI backend, so nothing out of the ordinary). While that's somewhat encouraging, the whole local experience was still far from pleasant with all the noise and heat.

[0] https://deepclause.substack.com/p/how-to-make-small-models-p...

Mathematics has always been an experimental science to some extent. While Newton, Euler and Gauss would spend a lot time calculating numerical approximations by hand, modern mathematicians have been doing the same using computers and software. And once an a clear picture emerge about what’s going, you can start to formalize that and attempt to prove and communicate your results in the standard definition, proposition, lemma, theorem scheme. (Btw there is even a journal called Experimental Mathematics devoted to this approach).

I don’t see that LLMs will fundamentally change this, but rather accelerate the speed of mathematical research.

Some computer generated proofs might of course be hard to understand, but at least their existence gives another data point work with.

Doing Mathematics is more than proving something, that’s just the end of a long road spent pondering at one’s desk about how things could work out.

> This trend dates back at least to the machine learning bible, Elements of Statistical Learning (2001).

Could you elaborate on this?