ponyous

That's very well put. I noticed the issue myself when building my own evals and naturally started drifting into the direction you are describing, but you describe it so succinctly that now I am seeing how it should actually be done. Thank you

One of the first very cool demos I've seen of ChatGPT was in a post where the author prompted AI "pretend to be a terminal". Then went on to execute commands like `cd .fbi`, `ls files`, `cat files/confidential.txt`. Very fun demo. Looking forward to exploring similar idea with this terminal

Organization. I am always lost. Gitlab pipelines seem much more straigh forward to understand for me.

As someone who’s been building AI for CAD[0] with OpenSCAD. I’m very thankful to OpenSCAD maintainers it’s been a life saver. Terse syntax, a lot of plugins and great CSG kernel.

People keep saying I should use other kernels for fillets and stuff, but from my testing AI is just not as good with, for example build123d as it is with OpenSCAD.

[0]: grandpacad.com

Was curious how the race looks like if you look at Pareto frontier of LLM capability since this hype started. So I asked Claude to generate me a video. And it's quite insane how good it is and I thought you guys might find it interesting too.

How did you connect it to search console and do you like the integration? I would like to do this too, and eventually to Google Ads too.

It really depends on the complexity of the model you are trying to generate and the budget you have.

You mean people blind in one eye? I think it still works, but obviously depth perception will not be there

Surprisingly Gemini models. Spatial understanding seems to be the best for the price and speed.

https://grandpacad.com - AI modeling for 3D printing Been at it for about year and a half.

Really exciting stuff is happening literally every month, because underlying models are getting better and better. When I started it was pretty basic: “make a cube with a hole through it”. Now it’s at the point “make a raspberry pi 4 case” and the agent searches, builds, verifies…

What surprised me in this process is how little meaning AI benchmarks have. Pareto frontier for my use case looks completely different than any other benchmark portrays.

> I have a feeling that the price of 1 million tokens transmitted over the internet is more expensive than cache hit.

And this kinda makes sense. What is cheaper few KB of disk space or internet bandwidth?

We went for 16% intelligence bump according to artificial analysis for +82% of the cost. Interesting.

Comparing 4.8 Opus with Fable 5.1

On my benchmark where AIs generate ~20 different 3D models about 1/2 the time of Opus and 1/3 of the time of Kimi K3 and 2/3 of time of sonnet.

Yeah you can absolutely do multiple parts, although the mating features are still a bit rough you can do a lot already.

GrandpaCAD - AI 3D modeling software focused on simplicity. Made it so even my grandpa could model. He’s been asking me for years when will I teach him how to 3D model. I tried, we failed and then I seen him use ChatGPT so I knew there was a better way than traditional CAD tools.

Recently we also got European funding and the project got some traction. Very exciting times ahead.

https://grandpacad.com

You are right, relatively to other llm providers this is not slow. But if you think what is possible when you have 1000t/s a sec you might find it slow.

Spark email client supports this. It’s great

I agree with the conclusion and am happy to see this blog post, but this killed a bit of credibility for me:

> Using a single LLM judge for scoring. Every score here comes from one model, GPT-5.6 Luna, looking at one image at a time. I didn’t do much alignment and didn’t check how often it agrees with itself on a re-run.

Having used a similar setup (with previous gen LLMs) to evaluate the 3D models that my product[0] generates, it turned out there was no correlation at all. LLM judgments were very much random and I assume judging SVGs is not that far from judging 3D models. I guess I have to re-test this with current gen.

[0]: https://grandpacad.com

Interesting way to show off a model last on every benchmark. Not sure any other lab is doing this

How can I become a trusted organization/partner? For my SaaS[0] where we generate 3D models using code it would be an absolute game changer to have such speedy generations. This would mean AI could do 10 iterations in the time it makes 1 now.

[0]: GrandpaCAD.com

As someone who just had success with equivalent system in EU my recommendation is to get someone who's done it before to do it for you. I hired an agency. They took 8% fee, which is pretty low, usually it's between 10% and 15%.

Absolutely. Running it now, will update this comment in about 30 mins.

Edit: Surprisingly very good results with 3.0 flash with high thinking.

Cost: $0.06

Duration: 3.22 min

Code Errors: 1.3 per attempts (meaning on average it had to retry 1.3 times)

Adherence was on par with 3.5 flash Low thinking

I don't have the eval results live yet, so I cannot share them yet.

I was benchmarking using a soon to be released new version of my AI CAD modeling software[0]. It's basically an agent that has access to tools that can execute build123d scripts, get sculpted models, blender to combine sculpts + parametric models, tools to inspect the model (visually and with code), search datasheets, ...

I tried what you recommend a while ago (asking an AI to evaluate using different angles) and the AI evaluations were extremely bad - barely any correlation to what I scored. Things have gotten better, but I don't trust it enough yet.

Here is how I score adherence (and how AI did as well, but I tried methods where it would just give back a boolean "pass" or not):

    <0.2 → Poor – Misses core intent; largely irrelevant or incorrect.
    <0.4 → Weak – Partially relevant; significant omissions or errors.
    <0.6 → Fair – Covers main points but lacks completeness or precision.
    <0.8 → Good – Mostly accurate; minor gaps or deviations.
    <=1.0 → Excellent – Fully aligned; precise, comprehensive, and faithful to intent.

Here is the scenario list (prompts are much more detailed):

    dragon-bottle-stopper
    editing-param-mid-conv
    editing-parametric-enclosure
    editing-swap-material-param
    editing-text-edit-cube
    multi-turn-bird-house
    multi-turn-dice-tower
    multi-turn-modular-planter
    multi-turn-phone-stand
    multi-turn-shelf
    one-shot-bookend
    one-shot-cable-clip
    one-shot-chess-queen
    one-shot-coaster
    one-shot-coffee-cup
    one-shot-dog-tag
    one-shot-dragon-figurine
    one-shot-hex-bracket
    one-shot-keychain-fob
    one-shot-low-poly-tree
    one-shot-pegboard-hook
    one-shot-pi4-case
    one-shot-threaded-jar

[0]: https://grandpacad.com

Just ran and scored 63 3d model generations (via code) across high and no reasoning. 3D Modeling benchmark quickly shows spatial, logic and code performance of the model so I think it's a very good indicator of the quality.

Here are the results compared to Gemini 3.5 Flash:

    Model + config          CodeErr/gen   Cost/gen   Median time   Quality
    gemini-3.5-flash, low      0.71        $0.18        68s       baseline
    GLM 5.2, reasoning high    0.61        $0.18       289s         -6.0%
    GLM 5.2, reasoning off     1.52        $0.10       126s        -13.6%

Although it is cheaper, it is significantly slower, and results are worse overall. Surprisingly - high reasoning produces less code errors than gemini 3.5 flash, but when I actually look at the models they are worse.

Edit: I recently ran evals with Kimi 2.7 and MiniMax-M3 and this is clearly open source SOTA model, by far.

Basically double from Opus 4.8 IIRC

> Surprisingly written by a human :)

Article ends with this

Bun is(was?) a lot about performance. How does it compare to zig?

I've run a tons of benchmarks for OpenSCAD for all kinds of models and setups, and what I realised is:

- Models are very jagged (might excel in one type of 3d model, but not another)

- Gemini models are the least jagged in my experience and have the best image understanding

- Gemini models are also the most creative (which may be undesirable if you want precise CAD part)

- Overall this benchmark doesn't prove much because one 3d model (and one attempt) is just not enough. I am usually testing on at least a dozen models each generated 3 times, but should really do much more, but it's too pricey for a solo dev.

Still, thanks for publishing this. Will be definitely run flash 3.5 soon to see how it performs.

I've seen this and other attempts like this[0] while exploring improvements for my CAD AI[1]. And I think these are potentially powerful solutions, but none of the current projects/weights have enough training (data or time) to make it work on arbitrary models. MeshCoder pretty much works only on models based of their training data. I haven't tried GenCAD but other commenters have confirmed my suspicion.

[0]: https://daibingquan.github.io/MeshCoder/

[1]: https://grandpacad.com