guregodevo

The gain comes from checking and restarting, not from reasoning. This harness is built using Mario orchestration which works pretty much like any DAG orchestrator. It's ideal for long repeatable thinking tasks.

For this benchmark i 've been using OpenRouter. If you have enough RAM or you can ssh a cloud shell this intro https://memdoor.ai/docs/open-weight can work too. I have reached with the ssh door a Google Cloud Box where the 27B model ran with an L4 g2-standard-16. A inference engine server is setup easily with memdoor (https://github.com/guregodevo/janis) on any CPU or GPU box. Ideally i think we need a quorum of SSH boxes like this, distributed that you can ping with a mention such as @box:coder. It's WIP . Any suggestion for running a team of workers in remote machine is welcome. I'd like to find a cheap solution.

This is a benchmark that solves ten hard problems in five passes. It scores 9.8 against the authors' 9.6 and Fable 5's 8.0. Memdoor workflows are the harness that replicates the original GVS5H thread harness posted in HN last week. Raw results (i.e. each call's tokens, cost and host) are in the bench folder for verification.

Happy hallowen! ::::

Actually we tried Cerebras. It caps output at 32K tokens so the 128K reasoning (the author budget) didn't fit. Cerebras speed is real though

will definitively try it. Overall with a smarter model the improvement is in the token usage and latency. Tbh I think i could get a 10 score if i tuned the prompt and temperature a little more. The original author harness is already giving 9.6 score which is pretty good. However this harness loop can take more than 1 hour on the hardest problem at 100 token /sec. A smarter model such as 3.8-flash-next could get 10 score and resolve the hardest problem faster. That's my estimate i'll give it a try. Thx

Here the intent is to replicate GVS5H with memdoor a headless coding agent I created months ago. Anyone can reshape the harness by redefining the steps in yaml files , models and temperatures. Up to you guys to try you have the recipe in this repo.

I tried to compare with a workflow i built with this harness : https://github.com/guregodevo/memdoor/tree/main/examples/wor...

The main issue i found is that Qwen3.8-27B thinks a lot. On a hard problem the planner runs 32K tokens of reasoning and never writes the plan. Even with thinking off on Cerebras, a problem takes roughly 3 minutes and the 100 problems about 1.5 hours.

A worflow is basically a DAG of tasks that are defined in YAML files. Once the agent has built it it can be run retried and repeated. Think of workflows as a DAG in Airflow or any DAG orchestrator.

Pretty intuitive tool. I like it.