I have a personal benchmark suite that runs a bunch of difficult algorithmic problems in 3 tiers: vibe coded prompt (just a few sentences), vibe coded prompt plus a description on what the solution will be evaluated on, and finally a fully specc'd prompt that leaves little space for interpretation.
The current Fable models are approaching a level where the -eval runs perform systemicallty better than the "clear spec" runs. (And right now its just Fable 5.1 and Opus 5.5 that behave so well, all the other models I tested - all the OpenAI and the leading Chinese models - work better with clear specs). I'd encourage everyone to experiment a bit; I am running my serious workflows with labourious force-me-through-all-the-design-questions mode, whereas my hobby projects, some of them hard, I am trying to see how much I can get away with with vibe-grade prompts. My conclusion is, with Fable its starting to look like what Vibe coding was promissed to be a year ago - a magical box that generally does things very well.
Btw, congrats to the OP -- the game is enjoyable and I played a few levels. I dont remember when I last played any games for more than 5 minutes. Good stuff.
The APIs aren’t clean, which creates indirections to other parts of the codebase to check on leaking abstractions. It doesn’t learn, though, so you pay for those indirections every time (right now, anyways).
I’ve not been impressed with its ability to clean that up with a generic “make it gud” kind of prompt. I have to point it at something crufty, but it will fix it then
For work stuff, I don't dare 'vibing' anything, and force myself through torturous design cycles and regular refactoring runs. But thats serious stuff with hundreds of thousands of users.
Btw. Fable 5.2 seems to burn through tokens with gusto. I ran my benchmark suite on it, added a complex feature to a hobby project (a gothic cathedral designer), and voila, 2 resets gone.
I’ve noticed because I’ve been doing Clojure and turns are taking notably longer due to unit tests because it doesn’t really grok its own system anymore. It is like 20k LoC at this point with unit tests lol
Whether the "cruft" is test accumulation, multiple conflicting architecture approaches, a bazillion ADRs/"laws", or just pure code sprawl, the outcome is token and time waste which gets worse as it compounds. And it compounds quickly.
I've tried automated code review through multiple lenses, more ADRs, instructions out the wazoo, but at the end of the day I just cannot figure out how to replicate the "through-line" consistency that responsible humans who need to maintain a system generally create.
Of course, most software projects do eventually fall victim to this same problem and need to be rewritten, but it takes place on the order of years, and by the time you need a rewrite the business has changed enough to demand one anyway. But with AI-owned products, this can happen in days or months.
Most of the time, anyways.
It keeps me in the code enough to feel the friction of bad APIs but high enough that I’m not reviewing every line of code.
I also try to scope my reviews to only the intersections between systems (SQL schema, library APIs, CLI flags, etc).
I’ve got nothing on the tests. Mine writes a million of them. Some are brilliant, some are stupid.
Like it once did TDD by storing transformations to the same base object in a list rather than specifying the whole object. Clever.
The very next test was an arity test to make sure it threw errors if you gave the function the wrong number of arguments. In a webapp, not a compiler or something. Stupid. Do not unit test the language’s runtime lol
The process of using AI to create things changes how you evaluate the quality of what's generated, so really the answer is "until you accept the output first time."
That's a combination of AI improving and your standards changing.
Or you'll never get something you're satisfied with...
To answer your question : I don't know, but probably not in the foreseeable future.
I'd be more specific and state that these models are now way far from being LLMs and instead they are full blown agent orchestration systems capable of running multi-layer ReAct loops and tool-calling, and with your whole session history as context. It doesn't make any sense to refer to them as glorified chatbots as the LLM part is nowadays far from being the component that delivers the highest value.
With a LLM alone, all you have is a stateless chatbot. If you increase the parameters of the LLM by a few orders of magnitude, you still have a stateless chatbot.
If you start with an anaemic LLM and run it with a ReAct loop and tool-calling, you now have something you can use to do agentic coding and RAG-like capabilities.
There's a reason why relatively small open weights models are starting to perform on par and even outperform models that need a huge datacenter to run.
Yes. Feel free to check recent benchmarks from the likes of OpenRouter
Yes it is. The rest of the agent could basically be a while loop, a two sentence system prompt and a shell invocation, and it would still work reasonably well.
While it's nice that LLMs are getting better at arriving at the desired deliverable with fewer surprises (or broken things) doing it is not necessarily a sign that the model is actually smarter, just an indication that what you are doing is likely a well-trodden path for the model, the RL and the harness.