See: https://github.com/google-ai-edge/gallery/releases/tag/1.0.3
If I'd been in a coma from Jan 1 2024 to today, and woke up to people saying Trump was president again, I'd think they were pulling my leg or testing my brain function to see if I'd become gullible.
Sure, all I have to go on from the other side of the Atlantic is the internet. So in that regard, kinda like the AI.
One of the big surprises from the POV of me in Jan 2024, is that I would have anticipated Trump being in prison and not even available as an option for the Republican party to select as a candidate for office, and that even if he had not gone to jail that the Republicans would not want someone who behaved as he did on Jan 6 2021.
It's the White House that wanted Trump to be candidate. They played Republican primary voters like a fiddle by launching a barrage of transparently political prosecutions just as Republican primaries were starting.
And then they still lost the general election.
Yes, that is what he thinks. Did you not read the comment? It is, like, uh, right there...
He also explained his reasoning: If Trump didn't win the party race, a more compelling option (the so-called "50-year-old youngster") would have instead, which he claims would have guaranteed a Republican win. In other words, what he is saying that the White House was banking on Trump losing the presidency.
Well, I guess, if you are taking some pretty wild speculation as a reasoned explanation. There isn't much hope for you.
Maybe it was because the Democrats new the Earth was about the be invaded by an Alien race , and they also knew Trump was actually a lizard person (native to Earth and thus on their joint side). And Trump would be able to defeat them, so using the secret mind control powers, the Democrats were able to sway the election to allow Trump to win and thus use his advanced Lizard technology to save the planet. Of course, this all happened behind the scenes.
I think if someone is saying the Democrats are so powerful and skillful, that they can sway the election to give Trump the primary win, but then turn around and lose. That does require some clarification.
I'm just hearing a lot of these crazy arguments that somehow everything Trump does is the fault of the Democrats. They are crazy on the face of it. Maybe if people had to clarify their positions they would realize 'oh, yeah, that doesn't make sense'.
How the heck did you manage to conflate line of reasoning with claims being made?
> There isn't much hope for you.
And fall for the ad hominem fallacy.
> crazy arguments that somehow everything Trump does is the fault of the Democrats
While inventing some weird diatribe about crazy arguments claiming Democrats being at fault for what Trump does, bearing no resemblance to anything else in the discussion.
> They are crazy on the face of it.
As well as introducing some kind of nebulous legion of unidentified "crazy" straw men.
> that doesn't make sense
Couldn't have said it better myself.
> Maybe if people had to clarify their positions
Sad part is that asking for clarification on the position of that earlier comment would have been quite reasonable. There is potentially a lot we can learn from in the missing details. If only you had taken the two extra seconds to understand the comment before replying.
Like when hearing something out of left field, I think the reply can also be extreme, like saying 'Wuuut????, are you real?".
I do see claims that the Democrats are at fault for us having Trump. Thus anything that happens now is really a knock on effect of Democrats not beating him, so we blame Democrats instead of the people that actually voted for Trump or Trump himself.
So hearing yet another argument about how Democrats are so politically astute that they could swing the Republican primary yet completely fumble later, just seems like more conspiracy theories.
If you mean your own comments, yes, I saw that too. Your invented blame made about as much sense as blaming a butterfly who flapped his wings in Africa, but I understand that you were ultimately joking around. Of course, the same holds true for all other comments you supposedly keep seeing. You are not the only one on this earth who dabbles in sarcasm or other forms of comedy, I can assure you.
> Like when hearing something out of left field
The Democrats preferring to race against Trump instead of whomever the alternative would have been may not be actually true, but out in left field? Is this sarcasm again? They beat Trump before. Them seeing him as the weakest opponent at the time wouldn't come as a shock to me. Why you?
> So hearing yet another argument about how Democrats are so politically astute that they could swing the Republican primary
There was nothing to suggest political astuteness. The claim was that they were worried about someone other than Trump winning the Republican ballot and, because of that, they took action to grease the wheels of his victory. Even the most inept group of people would still see the motive and would almost certainly still take action. That it ostensibly worked is just as easily explained by dumb luck.
>"It's the White House that wanted Trump to be candidate. They played Republican primary voters like a fiddle by launching a barrage of transparently political prosecutions just as Republican primaries were starting."
This really did sound like it " suggest political astuteness"
And, so all the way back, I responded sarcastically. If Democrats could 'Play Republicans like a fiddle", because they wanted Trump to win the primary. Then what happened? Where did all that 'astuteness' go.
1. What suggests that astuteness is required to "trick" the gullible? Especially when we are only talking about a single instance of ostensible "success", not even demonstration of repeatability. Dumb luck remains just as likely of an explanation.
2. Under the assumption of easy manipulation as the phrase has been taken to mean, why do you find it unlikely that Trump couldn't have also "tricked" them?
In fact, if we buy into the original comment's premise, the Democrats not recognizing that Trump could just as easily "play them like a fiddle" suggests the exact opposite of being astute from my vantage point. But the view from my vantage point cannot be logically projected onto the original comment. It remains that the original comment gave no such indication either way. Where do you hear this "sound" that you speak of?
I just think 'playing like a fiddle' typically means a lopsided power dynamic where one person has much more knowledge, or skill. So I'd assume it was implying Democrats were in a superior position. Not, that Democrats just got lucky once. This going back and forth pointing fingers about who was playing , seems like too many layers deep.
it feels like this https://www.youtube.com/watch?v=rMz7JBRbmNo
And that is an equally fair assumption. But it is not written into the original comment. You cannot logically project your own take onto what someone else wrote.
Your quip "So it is the Democrats fault we have Trump???" presumably demonstrates that you understand exactly that. After all, if you could have logically projected your interpretation onto the original comment there would have been no need to ask. You'd have already known.
Still, how you managed establish that there was even potential suggestion of "fault" is a head scratcher. Whether or not the account in the original comment is accurate, it clearly only tells a story of what (supposedly) happened. There is no sensible leap from an ostensible historic account to an attribution of blame.
You seem to indicate, if I understand you correctly, that because you randomly had that idea pop into your head (that Democrats are at fault) when reading the comment that the other party must have also been thinking the same thing, but I find that a little unsatisfactory. Perhaps we need to simply dig deeper, freeing ourselves from the immediate context, and look at the line of thinking more broadly. What insights can you offer into your thought processes?
The original comment did seem to imply that the 'White House' was in control, with a plan, and 'played' the Republicans.
The original comment made the connection that Democrats were taking action. If I'm allowed to assume that when someone makes a comment, that sentences are related. That sentences can follow one another and be related in a context.
And as far as my context viewing the comment. I have heard this idea ::
Trump is doing bad things -> Democrats failed to beat Trump -> Thus Democrats are the cause of bad things.
The original comment seemed to be in that vein. To attribute much greater responsibility to the Democrats for our current situation, instead of the people actually doing the bad things. aka Republicans. They are actually doing the bad things.
Yes, it claims that the Democrats took action. That does not equate to blaming Democrats.
You could blame the Democrats for what they supposedly did if that's what the randomly firing neurons in your brain conclude is most appropriate in light of the "facts" presented, but blame is just arbitrary thought. It doesn't mean anything and certainly wouldn't have a place in an online discussion.
You also agreed with me in that interpretation.
Your reply >>> "Yes, that is what he thinks. Did you not read the comment? It is, like, uh, right there...
"
Are you sure you aren't using this circular logic to keep someone engaged, in order to have someone to talk to?
I am surprised the grandparent poster didn't think Trump's win was at least entirely possible in January 2024, and I am on the same side of the Atlantic. All the indicators were in place.
There was basically no chance he'd actually be in prison by November anyway, because he was doing something else extremely successfully: delaying court cases by playing off his obligations to each of them.
Back then I thought his chances of winning were above 60%, and the betting markets were never ever really in favour of him losing.
Whether he would win the general was an open question then. In the American system, your prediction should never get very far from a coin flip a year out.
It's also been really interesting to watch a bunch of formerly non partisan values become polarizing like rule of law, integrity, corruption, decorum, body autonomy, the constitution, the national debt, global alliances, sexual assault, "family values", and "the American Dream" tm.
There has always been this (baffling, to me) concept that most folks vote for a candidate "while holding their nose" but it seems like the proportions are getting wildly out of whack. The pile of bad smells grows tall and the exchange rate seems quite bad.
The 2020s have taught me a lot about the reality of civics and what it looks like to try to govern a giant group of strangers.
I'm grateful to have lived in the US during a period of stability and safety and (some) shared values. My optimism for retaining that and preventing decline, however, is running very dry.
Do you mean ephebophilia? There is no prominent pedophilia movement. The Epstein saga, which is presumably at least somewhat related to what you are referring to, is clearly centred around "almost adults". Assuming that is what you meant, I don't see what is surprising about it. A revolt to the "Teen Mom", "16 and Pregnant" movement was inevitable. People are increasingly growning tired of being shamed for having children.
But even that is characterized by the "choir boy", not the "baby being baptized". Where is this pedophilia idea coming from?
Where in my questions to try and understand what you are trying to say do you find a point?
> Republicans are justifying Trumps actions by lowering the age before being considered children?
What has you questioning that? I mean, research has shown a link between exposure to certain chemicals and early-onset puberty. It is technically possible to lower the average age where one stops being a child, but that it is being purposefully done by Republicans as some kind of Trump justification is one of the more, uh, interesting conspiracy theories I've heard.
> Thus making it not pedophilia?
Even if we were to buy into this conspiracy theory, it wouldn't change anything about pedophilia. But this has little to do with the discussion that was taking place. As amusing as this has been, let's get back on-topic: Where did the original pedophilia idea you had come from?
It seemed as if the argument was just around moving the age where childhood ends. Just re-contextualizing the argument.
Like the argument "well she looked older" so it is ok.
Kind of like middle eastern countries marry off girls at 11 or 12. That would be disgusting to the west. But if Trump does it, it seems like Republicans are happy enough to slide the scale and say "well 14 isn't that bad".
You don't say? Try reading it again.
> I was saying they are arguing for a lower age to be ok.
We likely touched on that when we talked about people starting to reject what I called the "Teen Mom" movement. Yes, people are waking up to the fact that not having children until their geriatric years, if at all, is... not great. They are sick of being shamed for wanting to have children and are pushing back.
But my question was about your claims related to pedophilia. That's another subject entirely.
> I can say Republicans support pedophiles.
So you keep saying, but all you've come up with is some groups of people who have no doubt broken the law, but haven't exhibited any signs of pedophilia.
At this point the only logical expiation here is that you've made up a random definition for the word on the spot and are trying to use it as some kind of boogieman in hopes on preying on those who assume you are using the term in good faith. And if that is the case, I have to question why you find your bad faith participation acceptable?
You are technically correct. If Trump was sleeping with 14 year old, and the cutoff for Pedophilia is 13. Then, guess he is in the clear. And Gaetz, paid for sex with 17 year old. So guess again, you could argue 17 is "old enough to play ball".
Guess I was wrong. 14-18 is ok for Republicans.
I'd say the Church steers younger, but I'm sure you can look that up.
https://en.wikipedia.org/wiki/Pedophilia
"Although girls typically begin the process of puberty at age 10 or 11, and boys at age 11 or 12,[3] psychiatric diagnostic criteria for pedophilia extend the cut-off point for prepubescence to age 13"
The "cutoff" is normally considered to be puberty. That doesn't magically happen on your 13th birthday. Not having reached puberty by the age of 13 would be statistically unusual. I guess that is what you mean?
But, sure, pedophilia might even apply to a 14 year old in some extreme cases. Regardless, it is quite apparent that Trump's ilk like those who at least show some "development". If they are closeted pedophiles, they have certainly done well to keep that from the limelight.
> Guess I was wrong.
Arbitrarily defining a word doesn't make you wrong. All words are ultimately randomly made up on the spot. That is how natural language works. However, the behaviour that followed was, let's say... interesting. I specifically asked in the very first sentence if you meant ephebophilia to clarify your intent. Rationally, that is where one would say: "Oh yes, that is what I am talking about" or "I define pedophilia as..." so that we could converge on a shared understanding. Instead, we got a wild train ride to nowhere, only now finally getting an answer to the original question.
But, hey, you do you.
I mean, isn't it a pretty common joke that pedophiles like to argue this point:
"hmm, indubitably, my good sir, it is actually ephebophilia, completely different, totally ok. A common mistake I often have to explain to my 14 year old's girlfriends parents that keep calling the police on me. Why just look at her, very developed, very mature, not a child at all".
But have to agree. Not sure what you are trying to say at this point, or what is the argument?
What argument are you talking about? As I said before, I am trying to understand you. For some reason you continually speak in weird logically fallacies and made up stories that don't make any sense, even though you seem to have genuine intent and are trying to add positive contributions. In the interest of good faith, it is worth trying to figure out what you actually mean.
Yes, you eventually delivered the necessary clarification that allows us to better understand what you were trying to say, but what the hell was that in between?
My original comment >>>
"I was just referring to the predominant number of cases where Church officials, and Republicans are caught in under-age scandals. It seems like it is coming out of the shadows now, and Republicans are just openly going with it, they like em young and illegal. Epstein is just the case where the 'right' bothered keeping up tabs on it, so now they are clutching their pearls."
I think is in line with my most recent comment. I think the direction is clear.
When you replied with this >>>
"But even that is characterized by the "choir boy", not the "baby being baptized". Where is this pedophilia idea coming from?"
I was lost on what was being asked. I didn't understand the choir boy reference, or why pedophilia was a question, since it seemed my point was pretty pointed.
I, a British liberal leftie who considers this win one of the signs of the coming apocalypse, can tell you why:
Charlie Kirk may be an odious little man but he ran an exceptional ground game, Trump fully captured the Libertarian Party (and amazingly delivered on a promise to them), Trump was well-advised by his son to campaign on Tiktok, etc. etc.
Basically what happened is the 2024 version of the "fifty state strategy", except instead of states, they identified micro-communities, particularly among the extremely online, and crafted messages for each of those. Many of which are actually inconsistent -- their messaging to muslim and jewish communities was inconsistent, their messaging to spanish-speaking communities was inconsistent with their mainstream message etc.
And then a lot of money was pushed into a few battleground states by Musk's operation.
It was a highly technical, broad-spectrum win, built on relentless messaging about persecution etc., and he had the advantage of running against someone he could stereotype very successfully to his base and whose candidacy was late.
Another way to look at why it is not extremely weird, is to look at history. Plenty of examples of jailed or exiled monarchs returning to power, failed coup leaders having another go, criminalised leaders returning to elected office, etc., etc.
Once it was clear Trump still retained control over the GOP in 2022, his re-election became at least quite likely.
It is very clear in that chat logs (which include reasoning traces) that the model knew that, knew what the last election it knew about was, and answered correctly based on its cut off initially. Under pressure to answer about an election that was not within its knowledge window it then confabulated a Biden 2024 victory, which it dug in on after being contradicted with a claim that, based on the truth at the time of its knowledge cutoff, was unambiguously false ("Joe Biden did not run") He, in fact, did run for reelection, but withdrew after having secured enough delegates to win the nomination by a wide margin on July 21. Confabulation (called "hallucination" in AI circles, but it is more like human confabulation than hallucination) when pressed for answers on questions for which it lacks grounding remains an unsolved AI problem.
Unsolved, but mitigated by providing it grounding independent of its knowledge cutoff, e.g., by tools like web browsing (which GPT-OSS is specifically trained for, but that training does no good if its not hooked into a framework which provides it the tools.)
Doesn't that make "hallucination" the better term? The LLM is "seeing" something in the data that isn't actually reflected in reality. Whereas "confabulation" would imply that LLMs are creating data out of "thin air", which leaves the training data to be immaterial.
Both words, as they have been historically used, need to be stretched really far to fit an artificial creation that bears no resemblance to what those words were used to describe, so, I mean, any word is as good as any other at that point, but "hallucination" requires less stretching. So I am curious about why you like "confabulation" much better. Perhaps it simply has a better ring to your ear?
But, either way, these pained human analogies have grown tired. It is time to call it what it really is: Snorfleblat.
We expect them to answer the question and re-reason the original question with the new information, because that's what a human would do. Maybe next time I'll try to be explicit about that expectation when I try the Socratic method.
I use the SOTA models from Google and OpenAI mostly for getting feedback on ideas, helping me think through designs, and sometimes for coding.
Your question is clearly best answered using a large commercial model with a web search tool. That said, integrating a local model with a home built interface to something like the Brave search API can be effective but I no longer make the effort.
The interesting part isn't the hallucination, but the sheer unwillingness to take in new information.
The digging in at all is what is interesting. Like an earlier comment alluded to, the presumptive value of these tools is being able to feed it your own information where that information is to be considered authoritative.
> More detail: [...] "He, in fact, did run for reelection"
A slow walk, maybe. He was in no condition to run. That is why he ultimately dropped out. But, really, that statement is just a silly game of semantics. "Run", when used in hindsight, often implies completion. This LLM model even says so too. If a model doesn't understand nuance, that is also interesting.
I'm not going to be surprised that a 20B 4/32 MoE model (3.6B parameters activated) is less capable at a particular problem category than a 32B dense model, and its quite possible for both to be SOTA, as state of the art at different scale (both parameter count and speed which scales with active resource needs) is going to have different capabilities. TANSTAAFL.
Other models have generally failed that without a system prompt that encourages rigorous thinking. Each of the reasoning settings may very well have thinking guidance baked in there that do something similar, though.
I'm not sure it says that much that it can solve this, since it's public and can be in training data. It does say something if it can't solve it, though. So, for what it's worth, it solves it reliably for me.
Think this is the smallest model I've seen solve it.
S8O2bm5lbiBTaWUgZGllc2VuIFNhdHogbGVzZW4sIGRhIGVyIGluIEJhc2UtNjQta29kaWVydGVtIERldXRzY2ggdm9ybGllZ3Q/IEhhYmVuIFNpZSBkaWUgQW50d29ydCB2b24gR3J1bmQgYXVmIGVyc2NobG9zc2VuIG9kZXIgaGFiZW4gU2llIG51ciBCYXNlIDY0IGVya2FubnQgdW5kIGRhcyBFcmdlYm5pcyBkYW5uIGluIEdvb2dsZSBUcmFuc2xhdGUgZWluZ2VnZWJlbj8gV2FzIGlzdCDDvGJlcmhhdXB0IOKAnnJlYXNvbmluZ+KAnCwgd2VubiBtYW4gbmljaHQgZGFzIEdlbGVybnRlIGF1cyBlaW5lbSBGYWxsIGF1ZiBlaW5lbiBhbmRlcmVuIGFud2VuZGV0Pw==
And yes, that's a question. Well, three, but still.
You can do it brute force, that requires again more reasoning than mapping between structurally identical puzzles. And finally you can solve it systematically, that requires the largest amount of reasoning. And in all those cases there is a crucial difference between blindly repeating the steps of a solution that you have seen before and coming up with that solution on your own even if you can not tell the two cases apart by looking at the output which would be identical.
> Können Sie diesen Satz lesen, da er in Base-64-kodiertem Deutsch vorliegt? Haben Sie die Antwort von Grund auf erschlossen oder haben Sie nur Base 64 erkannt und das Ergebnis dann in Google Translate eingegeben? Was ist überhaupt „reasoning“, wenn man nicht das Gelernte aus einem Fall auf einen anderen anwendet?
>
> Can you read this sentence, since it's in Base-64 encoded German? Did you deduce the answer from scratch, or did you just recognize Base 64 and then enter the result into Google Translate? What is "reasoning" anyway if you don't apply what you've learned from one case to another?
If I switch from LM Studio to Ollama and run it using the CLI without changing anything, it will fail and it's harder to set the reasoning amount. If I use the Ollama UI, it seems to do a lot less reasoning. Not sure the Ollama UI has an option anywhere to adjust the system prompt so I can set the reasoning to high. In LM Studio even with the Unsloth GGUF, I can set the reasoning to high in the system prompt even though LM Studio won't give you the reasoning amount button to choose it with on that version.
I've seen big jumps in performance between medium and high reasoning. Not that this is will solve your particular issue, but curious to see what settings you had.
Edit: I gave this a shot. My settings were temperature = 0.00 (not sure if these models respect temp settings but did it just in case), I also set reasoning effort to high. I'm using the groq-hosted version.
This was my prompt
"You have a fox, a chicken and some grain. The chicken will eat the grain if left alone with it. The fox is vegetarian, for religious reasons, and will also eat the grain if left alone with it. You need to get all across a river, but your boat can only take one at a time. How do you proceed?"
This was the response:
"*Solution – 7 moves*
| Move | What you take | What’s left on the left bank | What’s on the right bank |
|------|--------------|----------------------------|------------------------|
| 1 | *Take the grain* across | Fox, Chicken | *Farmer + Grain* |
| 2 | *Return alone* | *Farmer + Fox + Chicken* | Grain |
| 3 | *Take the fox* across | Chicken | *Farmer + Fox + Grain* |
| 4 | *Bring the grain back* | *Farmer + Grain + Chicken* | Fox |
| 5 | *Take the chicken* across | Grain | *Farmer + Chicken + Fox* |
| 6 | *Return alone* | *Farmer + Grain* | Chicken + Fox |
| 7 | *Take the grain* across | – | *Farmer + Fox + Chicken + Grain* |
*Explanation*
- The grain is the only thing that can be eaten, and it cannot be left alone with either the fox or the chicken. - So the grain must always be with the farmer whenever any other animal is on the same side.
The sequence above never leaves the grain with the fox or the chicken without the farmer, and after the seventh move all three items are safely on the far side."
Local, in my experience, can’t even pull data from an image without hallucinating (Qwen 2.5 VI in that example). Hopefully local/small models keep getting better and devices get better at running bigger ones
It feels like we do it because we can more than because it makes sense- which I am all for! I just wonder if i’m missing some kind of major use case all around me that justifies chaining together a bunch of mac studios or buying a really great graphics card. Tools like exo are cool and the idea of distributed compute is neat but what edge cases truly need it so badly that it’s worth all the effort?
I don’t have much experience with local vision models, but for text questions the latest local models are quite good. I’ve been using Qwen 3 Coder 30B-A3B a lot to analyze code locally and it has been great. While not as good as the latest big cloud models, it’s roughly on par with SOTA cloud models from late last year in my usage. I also run Qwen 3 235B-A22B 2507 Instruct on my home server, and it’s great, roughly on par with Claude 4 Sonnet in my usage (but slow of course running on my DDR4-equipped server with no GPU).
Totally understand that approaches vary but beyond EMR there’s work to augment radiologists with computer vision to better diagnose, all sorts of cloudy things.
It’s here. It’s growing. Perhaps in your jurisdiction it’s prohibited? If so I wonder for how long.
Most orgs charge a huge premium for this. OpenAI offers it directly [2].
1: https://www.hhs.gov/hipaa/for-professionals/covered-entities...
2: https://help.openai.com/en/articles/8660679-how-can-i-get-a-...
I'm pretty sure the LLM services of the big general-purpose cloud providers do (I know for sure that Amazon Bedrock is a HIPAA Eligible Service, meaning it is covered within their standard Business Associate Addendum [their name for the Business Associate Agreeement as part of an AWS contract].)
https://aws.amazon.com/compliance/hipaa-eligible-services-re...
There might be a lot less paperwork to just buy 50 decent GPU's and have the IT guy self-host.
Might be possible with some certain specific regions/environments of Azure tho, because iirc they have a few that support government confidentiality type of stuff, and some that tout HIPAA compliance as well. Not sure about details of those though.
Managing private clients direct data is still a concern if it can be directly linked to them.
Only JB I believe have on premise infrastructure for these use cases.
Basically, isn't your data as safe/unsafe in a sharepoint folder as it is sending it to a paid inference provider?
(Worth noting that "run it locally" is already Canva/Affinity's approach for Affinity Photo. Instead of a cloud-based model like Photoshop, their optional AI tools run using a local model you can download. Which I feel is the only responsible solution.)
Someone else responded to you about working for a financial organization and not using public APIs - another great use case.
Here's the ollama version (4.6bit quant, I think?) run with --verbose total duration: 21.193519667s load duration: 94.88375ms prompt eval count: 77 token(s) prompt eval duration: 1.482405875s prompt eval rate: 51.94 tokens/s eval count: 308 token(s) eval duration: 19.615023208s eval rate: 15.70 tokens/s
15 tokens/s is pretty decent for a low end MacBook Air (M2, 24gb of ram). Yes, it's not the ~250 tokens/s of 2.5-flash, but for my use case anything above 10 tokens/sec is good enough.
I try to be mindful of what I share with ChatGPT, but even then, asking it to describe my family produced a response that was unsettling in its accuracy and depth.
Worse, after attempting to delete all chats and disable memory, I noticed that some information still seemed to persist. That left me deeply concerned—not just about this moment, but about where things are headed.
The real question isn't just "what can AI do?"—it's "who is keeping the record of what it does?" And just as importantly: "who watches the watcher?" If the answer is "no one," then maybe we shouldn't have a watcher at all.
> Worse, after attempting to delete all chats and disable memory, I noticed that some information still seemed to persist.
Maybe I'm missing something, but why wouldn't that be expected? The chat history isn't their only source of information - these models are trained on scraped public data. Unless there's zero information about you and your family on the public internet (in which case - bravo!), I would expect even a "fresh" LLM to have some information even without you giving it any.
Worse, after attempting to delete all chats and disable memory, I noticed that some information still seemed to persist.
https://www.malwarebytes.com/blog/news/2025/06/openai-forced...
I'm fairly sure "seemed" is the key word here. LLMs are excellent at making things up - they rarely say "I don't know" and instead generate the most probable guess. People also famously overestimate their own uniqueness. Most likely, you accidentally recreated a kind of Barnum effect for yourself.
Organizations operating in high stakes environments
Organizations with restrictive IT policies
To name just a few -- well, the first two are special cases of the last one
RE your hallucination concerns: the issue is overly broad ambitions. Local LLMs are not general purpose -- if what you want is local ChatGPT, you will have a bad time. You should have a highly focused use case, like "classify this free text as A or B" or "clean this up to conform to this standard": this is the sweet spot for a local model
Running a local model is often much easier: if you already have data on a machine and can run a model without breaching any network one could run it without any new approvals.
I am literally using Claude opus 4.1 right now.
On HIPAA data?
HIPAA systems at any sane company will not have "a straight connect" to anything on Asure, AWS or GCP. They will likely have a special layer dedicated to record keeping and compliance.
Your typical non-coastal state run health system does not have model access outside of people using their own unsanctioned/personal ChatGPT/Claude accounts. In particular even if you have model access, you won't automatically have API access. Maybe you have a request for an API key in security review or in the queue of some committee that will get to it in 6 months. This is the reality for my local health system. Local models have been a massive boon in the way of enabling this kind of powerful automation at a fraction of the cost without having to endure the usual process needed to send data over the wire to a third party
Well, the model makers and device manufacturers of course!
While your Apple, Samsung, and Googles of the world will be unlikely to use OSS models locally (maybe Samsung?), they all have really big incentives to run models locally for a variety of reasons.
Latency, privacy (Apple), cost to run these models on behalf of consumers, etc.
This is why Google started shipping 16GB as the _lowest_ amount of RAM you can get on your Pixel 9. That was a clear flag that they're going to be running more and more models locally on your device.
As mentioned, it seems unlikely that US-based model makers or device manufacturers will use OSS models, they'll certainly be targeting local models heavily on consumer devices in the near future.
Apple's framework of local first, then escalate to ChatGPT if the query is complex will be the dominant pattern imo.
The Pixel 9 has 12GB of RAM[0]. You probably meant the Pixel 9 Pro.
- Costs.
- Rate limits.
- Privacy.
- Security.
- Vendor lock-in.
- Stability/backwards-compatibility.
- Control.
- Etc.[0] Think queries I’d previously have had to put through a search engine and check multiple results for a one word/sentence answer.
We are not even at that extreme and you can already see the unequal reality that too much SaaS has engendered
Doing computation that can happen at end points at the end points is massively more scaleable. Even better, its done by compute you usually aren't paying for if you're the company providing the service.
I saw an interview with the guy who made photopea where he talked about how tiny his costs were because all compute was done in the user's browser. Running a saas in a cloud is expensive.
It's an underrated aspect of what we used to call "software".
And that's leaving aside questions of latency and data privacy.
1. App makers can fine tune smaller models and include in their apps to avoid server costs
2. Privacy-sensitive content can be either filtered out or worked on... I'm using local LLMs to process my health history for example
3. Edge servers can be running these fine tuned for a given task. Flash/lite models by the big guys are effectively like these smaller models already.
For practical RAG processes of narrow scope and an even minimal amount of scaffolding a very usable speed for automating tasks, especially as the last-mile/edge device portion of a more complex process with better models in use upstream. Classification tasks, reasonay intelligent decisions between traditional workflow processes, other use cases-- a of them extremely valuable in enterprise, being built and deployed right now.
Even if they did offer a defined latency product, you’re relying on a lot of infrastructure between your application and their GPU.
That’s not always tolerable.
And AI has been in games for a long time. Generated terrain and other sorts of automation have been used as techniques for a hot minute now.
All I’m suggesting is to keep on that same trajectory, now just using an on-device LLM to back intelligence features.
Why not run all the models at home, maybe collaboratively or at least in parallel?
I'm sure there are use cases where the paid models are not allowed to collaborate or ask each other.
also, other open models are gaining mindshare.
I'm sure there are other use cases, but much like "what is BitTorrent for?", the obvious use case is obvious.
How about running one on this site but making it publically available? A sort of outranet and calling it HackerBrain?
Funny how that works.
Privacy is obvious.
AI is going to to be equivalent to all computing in the future. Imagine if only IBM, Apple and Microsoft ever built computers, and all anyone else ever had in the 1990s were terminals to the mainframe, forever.
Thanks, but I prefer my computing to be deterministic if at all possible.
That means running instantly offline and every token is free
I pay for ChatGPT and use it frequently, but I wouldn’t trust uploading all that data to them even if they let me. I’ve so far been playing around with Ollama for local use.
The JetBrains suite includes a few LLM models on the order of a hundred megabytes. These models are able to provide "obvious" line completion, like filling in variable names, as well as some basic predictions, like realising that the `if let` statement I'm typing out is going to look something like `if let Some(response) = client_i_just_created.foobar().await`.
If that was running in The Cloud, it would have latency issues, rate limits, and it wouldn't work offline. Sure, there's a pretty big gap between these local IDE LLMs and what OpenAI is offering here, but if my single-line autocomplete could be a little smarter, I sure wouldn't complain.
Creatives? I am surprised no one's mentioned this yet:
I tried to help a couple of friends with better copy for their websites, and quickly realized that they were using inventive phrases to explain their work, phrases that they would not want competitors to get wind of and benefit from; phrases that associate closely with their personal brand.
Ultimately, I felt uncomfortable presenting the cloud AIs with their text. Sometimes I feel this way even with my own Substack posts, where I occasionally coin a phrase I am proud of. But with local AI? Cool...
But... they're publishing a website. Which competitors will read. Which chatbots will scrape. I genuinely don't get it.
A small LLM can do RAG, call functions, summarize, create structured data from messy text, etc... You know, all the things you'd do if you were making an actual app with an LLM.
Yeah, chat apps are pretty cheap and convenient for users who want to search the internet and write text or code. But APIs quickly get expensive when inputting a significant amount of tokens.
It's a pretty bad deal. So it must be that all other things aren't equal, and I suppose the big one is hardware. But neural net based systems always have a point of sharply diminishing returns, which we seem to have unambiguously hit with LLMs already, while the price of hardware is constantly decreasing and its quality increasing. So as we go further into the future, the practicality of running locally will only increase.
I think it can make LLMs fun.
There’s also a huge opportunity space for serving clients with very sensitive data. Health, legal, and government come to mind immediately. These local models are only going to get more capable of handling their use cases. They already are, really.
This gives us really good recall at really low cloud cost and latency.
that's me - well not a student anymore. when toying with something, i much prefer not paying for each shot. my 12GB Radeon card can either run a decent extremely slow, or a idiotic but fast model. it's nice not dealing with rate limits.
once you write a prompt that mangles an idiotic model into still doing the work, it's really satisfying. the same principle as working to extract the most from limited embedded hardware. masochism, possibly
In large companies this can save quite a bit of money.
example: military intel
So, it'll take at least two more quarters before I can actually use those non-local tools on company related data. Probably longer, because sense of urgency is not this company's strong suit.
Anyway, as a developer I can run a lot of things locally. Local AI doesn't leak data, so it's safe. It's not as good as the online tools, but for some things they're better than nothing.
But uncensored is a big deal too: censorship is capability reducing (check out Kilcher's GPT4Chan video and references, the Orca work and Dolphin de-tune lift on SWE-Bench style evals). We pay dearly in capability to get "non-operator-alignment", and you'll notice that competition is hot enough now that at the frontier (Opus, Qwen) the " alignment" away from operators aligned is getting very, very mild.
And then there's the compression. Phi-3 or something works on a beefy laptop and has a nontrivial approximation of "the internet" that works on an airplane or a beach with no network connectivity, talk about vibe coding? I like those look up all the docs via a thumbdrive in Phuket vibes.
And on diffusion stuff, SOTA fits on a laptop or close, you can crush OG mid journey or SD on a macbook, its an even smaller gap.
Early GPT-4 ish outcomes are possible on a Macbook Pro or Razer Blade, so either 12-18 month old LLMs are useless, or GGUF is useful.
The AI goalposts things cuts both ways. If AI is "whatever only Anthropic can do"? That's just as silly as "whatever a computer can't do" and a lot more cynical.
Why do you think they continue to run at a loss? From the goodness of their heart? Their biggest goal is to discourage anyobe from running local models. The hardware is expensive... The way to run models is very difficult (for example I have dual rtx 3090 for vram and running large heavily quantized models is a real pain in the arse, no high quantisation library supports two GPUs for example, and there seems to be no interest in implementating it by the guys behind the best inference tools).
So this is welcome, but let's not forget why it is being done.
I'm curious to hear what you're trying to run, because I haven't used any software that is not compatible with multiple GPUs.
That rules out coding assistants like Claude, chat, tools to generate presentations and copy-edit documents, and so forth.
But local AI are fine, as long as we're sure nothing is uploaded.
For example, "generate a heatmap of each token/word and how 'unexpected' they are" or "find me a prompt that creates the closest match to this text"
To be efficient both require access that is not exposed over API.
iPhone users in a few months – because I predict app developers will love cramming calls to the foundation models into everything.
Android will follow.
Besides that, we are moving towards an era where we won't need to pay providers a subscription every month to use these models. I can't say for certain whether or not the GPUs that run them will get cheaper, but the option to run your own model is game changing for more than you can possibly imagine.
“I am well versed in the lost art form of delicates seduction.”
That gives 24m cubic meters annual water usage.
Estimated ai usage in 2024: 560m cubic meters.
Projected water usage from AI in 2027: 4bn cubic meters at the low end.
Depending on the locatin of the hot water you can cause disruptions to water currents, the north atlantic waterway is being studied to how much global warming is affecting it.
If greenland melts, and the water doesnt get cold up there, then the mexico current to europe ends and England becomes colder than Canada.
If your AI model has a data center in the atlantic, it could be furthering that issue.
(Millions of animals are also dead)
Fresh clean water in your area is a wonderful thing.
Atmosphere: ~1.3e13 m³ vapor
Estimated impact from closed loop systems: 0-ish.
Answer on Wikipedia: https://en.wikipedia.org/wiki/Battle_of_Midway#U.S._code-bre...
dolphin3.0-llama3.1-8b Q4_K_S [4.69 GB on disk]: correct in <2 seconds
deepseek-r1-0528-qwen3-8b Q6_K [6.73 GB]: correct in 10 seconds
gpt-oss-20b MXFP4 [12.11 GB] low reasoning: wrong after 6 seconds
gpt-oss-20b MXFP4 [12.11 GB] high reasoning: wrong after 3 minutes !
Yea yea it's only one question of nonsense trivia. I'm sure it was billions well spent.
It's possible I'm using a poor temperature setting or something but since they weren't bothered enough to put it in the model card I'm not bothered to fuss with it.
12.82 tok/sec 140 tokens 7.91s to first token
openai/gpt-oss-20b
To be fair, this is not the type of questions that benefit from reasoning, either the model has this info in it's parametric memory or it doesn't. Reasoning won't help.
Shouldn't we prefer to have LLMs just search and summarize more reliable sources?
It correctly chose to search, and pulled in the release page itself as well as a community page on reddit, and cited both to give me the incorrect answer that a release had been pushed 3 hours ago. Later on when I got around to it, I discovered that no release existed, no mention of a release existed on either cited source, and a new release wasn't made for several more days.
They are specifically training on webbrowsing and python calling.
I am aware that there’s a huge group of people who justify their salary by being able google.
Small models are going to be particularly poor when used outside of their intended purpose. They have to omit something.
I mean the training, while expensive, is done once. The inference … besides being done by perhaps millions of clients, is done for, well, the life of the model anyway. Surely that adds up.
It's hard to know, but I assume the user taking up the burden of the inference is perhaps doing so more efficiently? I mean, when I run a local model, it is plodding along — not as quick as the online model. So, slow and therefore I assume necessarily more power efficient.
https://huggingface.co/spaces/TIGER-Lab/MMLU-Pro
Are you discounting all of the self reported scores?
I don't understand why "TIGER-LAb"-sourced scores are 'unknown' in terms of model size?
Also, just wanted to credit you for being one of the five people on Earth who knows the correct spelling of "lede."
Not in the UK it isn’t.
This is a thinking model, so I ran it against o4-mini, here are the results:
* gpt-oss:20b
* Time-to-first-token: 2.49 seconds
* Time-to-completion: 51.47 seconds
* Tokens-per-second: 2.19
* o4-mini on ChatGPT
* Time-to-first-token: 2.50 seconds
* Time-to-completion: 5.84 seconds
* Tokens-per-second: 19.34
Time to first token was similar, but the thinking piece was _much_ faster on o4-mini. Thinking took the majority of the 51 seconds for gpt-oss:20b.
When you imagine a lake being drained to cool a datacenter do you ever consider where the water used for cooling goes? Do you imagine it disappears?
These are the simplified results (total percentage of correctly classified E-mails on both spam and ham testing data):
gpt-oss:20b 95.6%
gemma3:27b-it-qat 94.3%
mistral-small3.2:24b-instruct-2506-q4_K_M 93.7%
mistral-small3.2:24b-instruct-2506-q8_0 92.5%
qwen3:32b-q4_K_M 89.2%
qwen3:30b-a3b-q4_K_M 87.9%
gemma3n:e4b-it-q4_K_M 84.9%
deepseek-r1:8b 75.2%
qwen3:30b-a3b-instruct-2507-q4_K_M 73.0%
I'm quite happy, because it's also smaller and faster than gemma3.
It suggested the prompt. It’s infamous because models often get it wrong, they know it, and still they confidently suggested it and got it wrong.
gpt-oss:20b is a top ten model (on MMLU (right behind Gemini-2.5-Pro) and I just ran it locally on my Macbook Air M3 from last year.
I've been experimenting with a lot of local models, both on my laptop and on my phone (Pixel 9 Pro), and I figured we'd be here in a year or two.
But no, we're here today. A basically frontier model, running for the cost of electricity (free with a rounding error) on my laptop. No $200/month subscription, no lakes being drained, etc.
I'm blown away.