JamesSwift

> Similarly this is just my opinion as well. I enjoy a good discussion here and I hope you are as well. Positive intents, even if my words my be curt, at times.

I have no issues with this thread : D But back to the discussion...

Again, I elided a lot of thoughts on the whole situation, but the other factor in my belief that this is an inevitability is that the SOTA labs churn too fast in both model behavior but also harness behavior. Normie businesses of the future are not going to hitch their wagon to a moving-target of both behavior and enforcement vs just pinning the equivalent of "opus 4.5" from a run of the mill inference provider. It just doesnt make business sense to be at the whim of a model update. And the entire perpetual motion machine aspect of the SOTA financial outlook _necessitates_ constant churn. If they dont release, they die.

> So where does that leave us? Nobody makes any money?

I think this leaves us with SOTA labs pushing hard for protectionist regulation to essentially require federal blessing to be an inference provider. I also think we will see a _ton_ of dark money poured into research to try and show things like theoretical smuggling/side-channel exfil attacks on models/model weights so that the SOTA companies can paint the _open models and model weights themselves_ as a bogeyman. If they can show that theres potentially latent attack vectors in those paths then that will then prevent all the "middle class" companies from being able to use them, especially if they need to adhere to eg SOC2/HIPAA certification etc

I dont think electricity cost is what is keeping John Doe the developer from running local models.

Sure, in an ideal Chinese-world, Chinese companies would not be open sourcing their models and weights. But I think its very clearly a deliberate effort to cut off the US-companies at the knees by doing so, as they are unable to compete on the marketplace directly due to the current business/political climate. I also dont think these need to be better than SOTA, just reach a "good enough" bar. Big businesses will continue to use the best-of-the-best, but that will not be the general public and also will not justify current financial outlooks (in my opinion).

> Most people don’t want to or have any interest in running or managing models locally

When I argue the hardware cost, I should note that I think this is the limiting factor for _improvement_ of these models. But ultimately I dont think most users will self host in that world. They will still pay for inference, but it will be for one of a dozen commoditized vendors who are able to reap the benefits of inference at scale.

> we all know Anthropic's revenue will not be anything remotely like flatline for awhile

The SOTA US labs are one chinese open model away from being abandoned overnight by the masses. Ive argued before, but argue again here, that the current hardware shortages are the main thing keeping this from being a reality currently. If more people were able to run local models for a reasonable all-in cost, we would have a much higher velocity of iteration and improvement in the open source community. But its still too niche while people cant justify the cost, and so they continue to pay the premium for inference.

> Most software, once mature, is mostly done.

I think this is a dated mindset with AI in the mix. Not only is AI making it easier to do big-bang rewrites which wouldnt have been done before, it is making working on brownfield codebases more feasible. It is lowering the barrier to entry for non-devs to contribute to codebases. Most software did not stagnate because there wasnt a need to grow it more, it stagnated because the friction and/or cost of change hit a critical mass.

I think we are much too early into the "new era" of software to understand what it looks like tomorrow, especially in terms of what we assume to be true today.

Umm, what? Did we just enforce a global codefreeze in perpetuity? Or, even easier, did we just enforce a global codefreeze for the linux operating system in perpetuity? No? Ok, then how do we end up with a finite number of serious security bugs?

Yes, this is a new normal. No amount of what we historically would have thought of as "normal" preparation can prepare you for being instantly 0-dayed across one or sometimes multiple simultaneous fronts. Merely mentioning something tangentially related to a 0-day is generally enough for any random joe to "clean room" recreate the 0-day and be off to the races.

The historic embargo system is likely not sustainable and we need a shift in both approach and security posture (ie fully trustless architectures by default). Not only that, but because of the insanely amplified frequency and risk of the 0-days, you generally have to throw the baby out with the bathwater and leave yourself completely vulnerable to supply chain attacks. Because not short-circuiting taking upstream updates leaves you open to attack.

Theres shades of the old normal in this stuff, but absolutely this is the dial being turned up to 11 and I dont know if theres any right answers for small teams out there.

Just ask it to build you one. The era of hyper-focused personal software is just beginning.

It just doesnt generally work how I would expect it to work until you learn the details of how to make it work how you expect. Especially as you get into RX it gets really confusing about which execution context you should be on. And execution contexts in mobile land are pretty convoluted to setup IMO.

So yes "async" is easy, but "multithreaded/parallel" is a lot more error prone and explicit.

EDIT: I should also note I havent been in the space for a few years so Im not fully current on how it looks today

Its async model is a pain point, but otherwise it really does a great job of being able to accommodate a vast range of use cases

Yes, there is constant drama but the ecosystem is healthy and you have a choice of multiple upstream "vendors" to hitch your wagon to in terms of your nix experience, and for the the most part they all compile down to the same set of hydra packages so its not really an issue.

Its much easier for the agent to look at a nixos module and decide that it should do `services.nginx.enable = true` vs reverse engineering everything it should do to install nginx on the system.

And likewise if theres an issue then its much more straightforward for it to unwind what it did in nixos vs a traditional software install and not accidentally mess with other users/services on the system.

Looks to be more falling with style to me

lol right? So youre telling me they shipping 150,000 features this week? Or, maybe, they whack-a-moled 149,995 perf or bug regressions and shipping 5 new things. :shrug:

Right, and yet the openai/anthropic models are still a tier above other options despite everyone gunning for them for at least one if not multiple years. While there are numerous jev-likes being spun up _in less than a month_ that are performing (as far as I can tell) up to and sometimes exceeding jev's accuracy.

This isnt the same as dropbox putting in a bunch of work to make the UX "just work", or even stripe doing the same for payments. This is "send me an api call that is in 1 of 3 forms, and get a response back in this form". The heavy lifting is in making that fast and cheap, and because they havent demonstrated a secret sauce in making it hard to outperform actual results that leaves a lot of room for competitors to get their existing models to run faster and cheaper for the same use case.

Dropbox: is I have to move my entire collection and setup the app on all my devices

Instagram: I have to move all my posts and also convince all my network to move over

Github: not a terrible example

If I decided to move off of jev tomorrow it would be an api key and a base api path update. Maybe 30 seconds of work.

I mean, a large majority of "pro jev" posts I see purely focus on the speed/cost and take their word on the accuracy. The entire product is "we guarantee the response will fit this schema, and will do it fast and cheap". Personally, I dont see how anything in this class can be "production ready" as a general classifier without the ability to fine tune on top.

35% 10% 55%

Great, cant wait for the inevitable "Are you sure you want to grant full disk access" prompts every week for eternity until the user is exasperated and revokes access.

Well in a lot of ways this demo actually is a mix of jev for simple stuff + a very intelligent harness to fill in the rest. Pay attention to the "jev counter" on when api calls actually happen and the kinds of decisions its making. Its very rarely in a tight loop, and when it is (eg in an item menu) it tends to randomly walk through options.

Its very cool, but the harness is doing a _ton_ of heavy lifting.

Sure it can, just ask it for next char or "done" in a loop

> I'm guessing, AI slots invading the store.

Well, same as my lack of sympathy for github and their "inability" to deal with "unprecedented AI generated traffic", I dont buy it. Seems really simple to setup an algorithm of "existing app from pre-AI is submitting another update gets put in the priority queue while everyone else gets the shared queue".

Sure it works fine because you havent yet experienced how much better it can be. I encourage you to stop after the 'plan' part, read the plan, and tweak it. Have part 1 of the plan be it coming up with a framework to evaluate success, part 2 be it spitballing solutions, and part 3 be taking the best solution and "productionizing" it. You will get much tighter output and the brainstorming wont all muddy each others context as it tries different solutions. Bonus points for having part 2 fan out on its own, though thats much harder on shared hardware.

> How do you all keep your context so low? Complex tasks just balloon it in my experience.

By breaking the problem into discrete steps and aggressively restarting the prompt from the current state after completion of said steps

"Whats new" is, I hate to break it to you, 99% AI-related. Its a pretty unique point in time here and likely will continue to be for at least 12-24 months. This is likely to go down as being as reality-altering as the internet but with the added wrinkle of things are able to progress/iterate _much_ faster. Not only are the models progressing at breakneck speed, but our understanding of how to use them effectively is as well. So theres legitimate "discoveries" happening every week.

You get other things like mutual exclusion, a clean baseline regardless of whats in the main directory, and potentially in the near future a way to tie into lifecycle hooks for eg creation/deletion [1]. That last part is nice in the new agentic AI era where you might want to setup/teardown local services that are isolated to those directories.

But yeah the filesystem cost is not ideal. I was just thinking about how to exclude my worktrees from timemachine backups automatically.

[1] https://lore.kernel.org/git/7c8b4673-37ac-45fa-ad8c-a1dc09af...

> Money is fungible and instantly redeeming rewards for a statement credit is almost always the optimal way to use credit card rewards.

Uhh I have no credit cards where redeeming for credit is the optimal play. Every single one has 'offers' that give a further multiplier on the dollar amount of the points. Eg redeem $80 of points for a $100 home depot gift card.

That seems narrow minded. Theres both "learnings directly related to the thing studied" and "learnings downstream from the thing studied". If you imagine mathematics being a huge sudoku puzzle of unknowns you are trying to fill in, each previously empty square you are able to fill in (or gain a smaller bound on) has implications in all sorts of other areas.

Time-to-fix is lower, but time-to-new-bug is also lower