The autotuner is the real thing - kernel-level search, config budget split by measured time share, winners cached per device/toolchain. Rotating resident weights across layers to dodge the hot-cache trap is a nice touch.
One question on model ranking: the fit scores look like predictions built from cost constants measured on a single M4 Max, not per-device measurements. How do you rank models across genuinely mixed hardware? Does the estimator improve from actual runs over time? That gap between predicted and measured fit is what eats mixed machine fleets alive.
Shameless plug since magnitude's goal is highly relevant to what i'm working on: teale.com - distributed inference across fleets of macs. If you're running local models on more than one box, check it out with your agent!
i have more agents than i have real humans who report to me now. as a serial founder/builder/domain hoarder, i have 5+ projects ongoing at all times and in the AI era, i can spin up a team of agents to work on them simultaneously.so at this point, across my businesses, i have some flavor of a gas city/company-wide ai harness for each along with instinct personally and it seems every SaaS app...
GLM-5.3-Flash at true 8-bit: 341 GB on disk, 328 GB resident, 288 experts across 46 layers, loads in 65 seconds. • 18.7 tokens/s generation, 35 tokens/s prompt, on a desk, on a $0 per-token bill. • Runs beside our whole agent city on one box with ~130 GB to spare. • Review test: caught 6 of 6 planted P1 defects, zero false positives, same score as the frontier model we pay for. • CRM test: 11 of 11 required records extracted, zero wrong writes, 45 minutes, first local model to clear the bar. • Serving a 131k-token window today; the model itself supports 1,048,576. Widened to 4 concurrent slots and still have 50gb+ of excess ram.
granted my cto still isn't moving all of our inference to glm5.3 but we've identified 40%+ that is currently handled by fable that we're routing locally instead and will do concurrent requests to verify/compare responses for a while.
would really appreciate testers but also any companies thinking about distributed inference powered by their own company devices on a private network. my own company has 200+ 16gb ram machines that we're using for inference.
Opensource/weight models will get better and better and eventually we will have mythos level running on smartphone/eyeglass hardware.
It is stupidly tedious currently to match supply with demand though because physical hardware like a 16gb ram MacBook doesn't mean there's truly 16gb available let alone matching models and all of their settings (kvcache, context limit, temperature, etc) to demand.
Would appreciate any help cus we need ai inference by the people for the people.
We're an outsourced accounting firm specializing in property management (appfolio, buildium, rentvine) that's building an open source ai harness/orchestrator for property managers.
$7M ARR, cash flow positive, looking to re-envision marketing/growth ai native.
email taylor at the apmhelp domain or comment and i'll reach out. =)
i'm working on github.com/teale-ai (distributed inference)
There's a ton of spam on ebay currently with configs of the 512gb ram variant going for sub $4k. Those are all scams/spam. You can tell instantly because the seller's account is/was created this month 2026.
The $20k+ listings aren't selling but what's interesting is technically, $20k for 512gb ram is still less than $40/gb ram on a single device. Compare that to a Nvidia spark with 128gb ram selling at $4k+ or if you're lucky, you snagged one for ~$3500. That's $31.25/gb
The retail on the 512gb was $9500 and after tax if you don't ship to a sales tax free state makes it only $20/gb ram.
I'm curious what price apple's newest studio variant will be priced at per gb ram but I recently bought a "refurb" from a reputable seller I've purchased many laptops from in the past at $13,750 which still put the 512gb at a $27/gb ram price point.
I'm bullish on local models especially due to gemma4. Good luck out there!
Already launched biz-in-a-box.org and a life-in-a-box.org spinoff as frameworks to replace every entity's QuickBooks. I'm using them myself for every project my agents are spinning up.
Stealth project is related to classpass but for another category of need that won't go away even in the age of AI that really is only possible with critical mass of supply to meet existing demand. Super excited cus there's no better time to build with unlimited agents that scale without people problems.
Lastly, can't wait to run local LLMs so no longer limited by tokens/money.
Fyxed deploys capital to landlords and their property managers for cash needs like repairs, turnovers, and improvements on rental properties.
We're at an inflection point where we have the deployment capital ($50M) to really take a big swing at this opportunity.
The perfect candidate has prior fintech (lending) experience, loves wrangling the complexities of reconciling/tracking money, and bonus points if you've built on top of Column.com's API.
Email me directly: taylor at fyxed dot com
We're integrating robots in and around buildings (focused in America for now). Example usecases:
- security guards, doormen, receptionists (low mobility using humanoids)
- security patrol around perimeter of buildings (high mobility using dogs)
- moving things/packages within a building (last step package delivery)
We are building on top of Unitree's hardware and there's lots to do.
- abstraction layer similar to Swift for iOS and Kotlin for Android
- appstore
- example reference apps (think mail, calculator, camera, maps)
996 work environment due to collaboration/communication requirements with folks in China and insatiable demand for deploying robots.
deploy at houmanoids.com goes to myself (founder) and cto/cofounder
We help manage 1 million+ residential rentals across America (primarily accounting, banking & finance) and we're likely to double our engineering team from 8 > 16 this year (broader co is 250+ FTE).
We have 3 engineering & product teams (US, Brazil, South Asia) and likely will prioritize folks in those regions.
Specific call outs for: - product folks with banking/fintech experience - financial/data analyst on hundreds of thousands of financials - crypto/dao lawyer
Please don't just send a resume but help us get excited about you by sending an email to "hn at apmhelp.com" the group is monitored by founder (me), CIO & CTO
Howdy! APM Help is silently one of the largest SFR (single family rental) operators in America with over 300k units we help manage. We see ourselves becoming the operations layer (back office) of every local (boots on the ground) property manager in America. We have big challenges we think can be solved with tech/automation/AI in the areas of leasing, accounting, maintenance, finance, insurance, and compliance because every single rental unit is effectively a small business.
We're effectively bootstrapped ($7M+ ARR), cash flow positive, operating close to the Rule of 60 and we're looking for engineers who are maybe tired of the hype and wanting to solve real fundamental problems in housing in America.
Email me directly with more than just a resume if interested! taylor at apmhelp dot com
Side note - we're very happy with international engineers but if you come to me asking for a US engineer comp, it takes away the primary reason why we'd consider international in the first place.