mesmertech

My thinking was models, esp Opus 5.5 can just hill-climb as long as you give them something achievable and a clear metric to improve on. So here's something I don't even have a clue how it exactly works.

The training itself happened in my basement 4070ti super over a day or so.

And yes I know its Opus slop, but still really cool how things I never thought possible unless you're an ML engineer is available to a run of the mill web-dev guy.

Thanks to this tweet from Sahil for even showing how the initial idea worked to distill Kokoro in the first place. https://x.com/sahilmdkr/status/2112212041045889409

Yea I can believe all these. I've personally have been having issues on these points:

"200 OK, no answer" - insane that openrouter's main feature is literally a fallback and streaming doesn't support 200 no content to fallback to another provider or smth.

"rate-limit by IP"... now it kinda makes sense why deepseek v4.1 flash rate limits me on prod but never seems to happen on local. Makes you have to basically pin Deepseek as provider, since I've never had 429 error on them

hopefully this come in openrouter api cause I'm not signing up for a specific provider's specific api platform, and have yet another thing that can bill me.

Also honestly if you have a thing that works, I'd say don't even share it. Its too easy to build a clone and start copying, which ofc was also possible a few years back but then it took actual effort and someone had to maintain it.

The only reason I'm sharing these is cause AI image generators are already like thousands in the market, so I mean if someone wants to go through the hell, I don't think its really that easy a market to succeed in anyways.

And here's my list of things that didn't work out: https://mesmer.tools/

Pretty basic ones, I got into AI images cause of that viral AI avatars thing that levelsio made 2-3 years back and have just been iterating on that idea(even though I think if I went into the AI text side of things, I'd have been much bigger at this point):

https://admakeai.com - Recently got this to 800 MRR , kinda trying to make it the main thing since the LTV on it is the highest. So if I could just figure out ads, this has the highest potential

https://aieasypic.com - First project that I made that made money from subs vs making money from ads, whole reason I continued to indiehack. Its kinda dying but core users still bring in 3-4k/mo , altho actual MRR is 1k/mo

https://bestphoto.ai - Made this after I got frustrated from the bad decisions I made on above, basically a tech rewrite + clone + simplification of aieasypic. Currently makes the most but at a lower margin than aieasypic. Not gonna say exact numbers, but lets say 5k+/mo

Announcement for the appearance of a commit, thats where we're at

Why are you guys not on Openrouter? I assume you'd get way more volume that way no?

Or does openrouter have like a specific contract you have to sign with them and requirements or smth? https://openrouter.ai/moonshotai/kimi-k3#providers

And on creativity at least visually, Gemini 3.1 pro is somehow still up there. But its really hindered by its inability to use tool calls effectively or make a long term plan.

My personal benchmark for new models has been to compare video making skills with something like remotion. Usually reveals if they have any "taste" or outside the box thinking.

I'm starting to not trust any "benchmarks" when it comes to frontier models at least. As an example Sol feels the most "gets stuff done" but has zero taste, or any capability to surprise.

And for frontier models I go one step ahead and try to recreate a complex animation video, with the ability for the model to review its own work. And at this Fable is still the top one. Ex: https://www.youtube.com/watch?v=uDAeAuYyl0E (recreation of Claude announcement video) and https://www.youtube.com/watch?v=cSsVNtGPOIg (recreation of a fireship video). Sol did something similar but you can instantly tell its AI slop from very small things, and it just has no narrative or thought put into the writing.

https://mesmer.tools/benchmarks/ai-video-generation , I usually put basic ones here.

Hadn't tested out any of the image models for training loras since Flux 1 dev, so I was curious which one is the current best. Results were pretty interesting

Models tested: Ideogram v4, Flux.1 Dev, Flux.2 Dev, Klein, Krea 2, Z-Image

I'm on max $200, I had this idea for around a month since Fable got banned. For the usage, not exactly sure since I had like 4 agent heavy tasks running at the same time.

I think personally you can do first half of this with the direction with fable and then render + checking using sol since thats what I ended up doing. Guesstimate I'd say it takes around 10-20% of the 50% fable usage

Had some fable usage to waste yesterday cause my reset was gonna happen. Ended up burning all my usage so had to finish the last part with Sol itself.

Imo Remotion based videos feel better to watch than just actual GenAI videos, like Seedance.

the exact prompt for reference:

"I wanna make a plan for making good fireship videos automatically. so my thinking is, you need to first need to find a good way to insert memes like gifs, short videos, images stuff like that. another part of a good fireship video is the voice itself, so you need to find a way to voice clone fireship's audio(probably download a sample video and use our existing higgs audio api, check airoleplay and search "higgs" for the api endpoint details and the api keys and stuff if you need). another part of it is the animations and creative text animations and extra touches that he puts, maybe for this you could analyze a couple of his vids in general using gemini 3.1 pro with openrouter in detail and you can ask multiple questions abut a video to explain in detail exactly what happens.

now remember you're the orchestrator and this is likely going to be a very long task so you need to be using your context very carefully, and assign various research tasks to opus or fable subagents depending on the complexity.

the video itself should be about claude fable 5 and make another one for gpt 5.6 sol. some more reference for how we recreated antoher vidoe with a single prompt before: '/Users/test/Documents/openmotion/recreate-video-user-prompts.md'"

As long as they have the best model they can afford to lose goodwill.

People who don't wanna spend too much on LLMs and are trying to optimize whats subsidized even on the Max plans are customers Anthropic is honestly better off without.

Ok thats a one month clock to the next Opus model at least, so thats a silver lining to a meh model.

Spoiler: more than half of it is just Meta ads spending cause its just addictive.

Seems really good at frontend work, and as a result on remotion programmatic videos. Not the best yet, thats still Gemini 3.1 pro(trained on actual videos) or Fable, but often better than what Opus can come up with

https://mesmer.tools/benchmarks/ai-video-generation

noticed it cause of ad manager, the main facebook site being down is kinda weird tho. I don't remember when was the last time that happened

Overall an improvement over Opus 4.8, but I'd still say Gemini 3.1 Pro has more of an artistic vision even tho it fails tool calls and writes buggy code sometimes.

Ik almost everyone is interested just in the SWE stuff, but this has been a good eval for me to think about how big the model is, how "creative" it is for generating new ideas etc.

More results from fable, with comparisons for Gemini, opus and some open source models: https://mesmer.tools/benchmarks/ai-video-generation

I think gpt 5.6 is coming out today so might wanna wait

/model claude-opus-4-8

seems to work but idk why they never set it so you can see it in the /model list.

"what model are you

I'm Claude Opus (claude-opus-4-8), running in Claude Code."

Yep sorry was just pulling it out my rear, not like a market trend that nearly every enterprise uses Anthropic or Openai models for coding or that Anthropic has had such ridiculous growth that they're 10x-ing year over year

My point was that even openrouter, the one place people who are looking for open source SOTA models go to, doesn't definitively have opensource models at the top. Esp considering quite a lot of the closed models usage is through AWS, GCP , Azure etc, probably dwarfing the usage on openrouter by a huge factor

As long as closed source is 6 months ahead in terms of current difference. Although this is hard to figure out using simple percent based coding benchmarks, you def. notice it when you're actually trying to do a long task. Even simple things like UI "taste" is enough for me to use opus instead of 5.5 though even though 5.5 is strictly better for anything that doesn't have a UI, ie backend, scripts, making agent workflows etc