pantelisk

Compute motion vectors, faux depth, and add DLLS5 to it maybe? Could be pretty cool, or pretty horrific. Only one way to find out :)

Brilliant work! Testing it now

LLMs have higher accuracy and are more versatile, but that comes with a cost (slower and more expensive to run). On the other end you can have your own mini classifier something built on top of ModernBert or DIET or even a hash classifier (these are narrow).

Models like JEV fall in the golden middle. Somewhat cheap, somewhat fast, somewhat general.

Many people have replied saying that narrow classifiers (eg bert) are better, and they are better in many ways (essentially free and faster). But... the real world is messy, in my experience it's actually very hard to make a really good classifier that will work across a specific niche domain especially when there is very fuzzy input. And the most interesting real world scenarios are nuanced and overtime there will be edge cases discovered where it fails and that will require retraining and re-evaluation etc etc. It can becomes a full project on its own that eventually collapses into a game of whackamole (improved in some direction but regressed elsewhere). Things like JEV (or similar models) provide ease of mind, just off load the complexity to it and move on to the next challenge type of thing.

Yes, the best way to think of a general classifier like this is like a smart switch statement. Essentially a "JEV" like thing becomes a sort of programming primitive. Once you see it, it's hard to not get excited.

But even if others surpass them and make better solutions, the fact that nobody was able to see it before typesafe is a testament of what they might be able to come up with next.

I sound like a fanboy but I swear I 'm unaffiliated with typesafe. I was building my own version of this way before they announced JEV (mine was ALE and it was mentioned here on HN for a bit), in use for VR gaming (so one can give commands to NPCs with voice and supports multiple commands in sequence in a single pass), but I missed the "killer usecase" of being a new primitive, like everyone else.

TLDR: There's a lot of value in thinking ahead and seeing the future. The clones are nice and exciting but they give me "I could have built this first, yes but you didn't" vibes. I hope they manage to keep it up and push the space forward again

I really have mixed feelings about this. Sure, emulation existed for a while but it always lagged enough or had small bugs to rarely match the spirit of the original (in rare cases it improved things with better framerates, resolution boosts and loading time cuts).

I guess this is somewhat late in the lifecycle of the PS5 but it feels different, like a death blow. Now that this is possible, it would most likely be easy to replicate when the PS6 comes out, or for any other console. Whatever competitive advantage with exclusives consoles have, goes away. And these deals cost a lot of money and take years to plan and execute.

I guess companies then transition to being software only, like Sega did? (though they never recaptured the charm they had when they were making their own hardware). Are we slowly witnessing consoles go the way of the cd player?

Same thing happened with the release of a certain programming language release a few weeks ago (that seemed fully vibecoded, which is fine - but was making multiple grandiose claims which attracted pushback). Multiple sock puppet accounts defending the poster religiously with attitude that is more reserved for reddit than HN.

It happens more around things related to vibecoding, it seems that influencer culture, empowered by this new tooling is starting to distort tech communities.

More and more of this will be happening. Before AI coding - the fact that something like photopea or pixelr existed, meant that somebody had to commit years of focused work to even reach to the same level of quality before starting to provide value and attract users. That alone made copying a losing proposition, or at least it raised the bar as it meant one had to be a truly exceptional and dedicated builder.

Nowadays it can happen in an evening, so everything successful will get copied and attacked and attempted to be displaced by noise and hype on certain internet circles alone. Like the cheap ridiculous clones of popular ios games and apps of the past, except now on an industrial scale!

I guess this is the new world we are entering, but can we at least keep the astroturfing and bad faith behavior away damnit?

I think what the parent poster is trying to convey, is that let's say the apocalyptic stuff has a 5% chance as you say, but the non-apocalyptic stuff (social-economic chaos, total centralization of power, eradication of social mobility, total information/trust collapse) might have a good 95% chance as we are seeing it starting to unfold already.

But the discourse is dominated by paper clip experiment discussions and not let's say by the fact that new grads have an unprecedented difficult time getting jobs. Unsurprisingly one of those is a sexy hypothetical beneficial to power and the other one is not.

Multiple problems can be important, pointing a different one out doesn't invalid or take away from another one.

Main difference is; this a desktop app, whereas audiomass is web.

In theory native apps have a higher ceiling for long processing tasks, since they can offload buffers on the disk and reuse - versus having to fit everything in browser memory.

No inference providers. It's all local, meaning all on your own device. The app writes no logs on your disk, so technically it's on-device + zdr?

Hey HN,

This is a follow up from a post of a few weeks ago about an AI gaming companion (an intelligent dog companion for Skyrim). I decided to double down on the local AI approach and package it as a self-contained creature, and try to broaden the types of hardware it can run on to.

I know it's a large ask to ask somebody to download a 9gb mystery executable, so I don't expect many to check it out. But in case you are curious.

It uses: - Optimized qwen-3-asr which has been converted for voice streaming (will open source this). - Quantized gemma4 models with custom megakernel inferrence + speculative decoding as the brain (I have a finetuned version for skyrim but gemma empirically worked better as generalzed) - Pocket-tts with added vector embeddings for emotion, optimized for CPU inf (https://pantel.is/projects/pocket-tts-raven/?b=1). - Threejs for the rendering of the terrarium/jar and the 2 sample game demos that come with it

At this stage it's more of an experiment art project local inference and intelligent NPCs than a polished product (especially with the whole gimmick of summoning him by speaking out an invocation during the intro. It’s still early alpha, but the idea was to make something you can spend a little time with, then leave to his own devices while you go about your day.

I think the doom demo uses a text representation of the world and it's basically, "projectile coming your way" -> "Strafe". "Enemy ahead" -> "shoot. So it works well when spawned in a room of enemies (as we see in the video).

If self driving is red means stop, green means go, and stay in your lane - then it would work great, but having to actually think and test which maneuver is optimal for a given situation while weighting safety, road rules, random unexpected actions and getting to your destination, I think it's a much bigger problem. A bigger model specifically trained on that maybe would do great, but then the output is not the constraint anymore.

But I haven't tried the model, so I 'm just ballparking and could be very wrong.

But how did they know about it? Did the agents independently discover it? Did the model know about it because this vuln was public somewhere and systems weren't patched yet?

Same for smartphones. There was a time of unlimited hype and secrery around new iphones. Who remembers the story of an iphone 5 prototype left a bar. Journalists were going crazy, people were signing petitions for Apple to not hunt down but instead forgive the employee that made such a grave mistake. People were offering millions to buy the prototype so they can brag they got the new phone 2 weeks before everyone else did.

Or who remembers the dancing disease of 1518, were people would stop what they are doing and start randomly doing the same dance. The lords? Out of their minds. The priests? Terrified the devil had taken hold of the flock! I have come to believe that it was probably some tik-tok like hype trend of doing a fortnite dance while waiting in line for bread and communion. And the energy back then, like now, was off the charts.

Hype and memetic trend seeking encoded deep in human psyche.

Audacity :) (as an app) and Audiomass on web (I am biased towards it of course)

It should look like a NES cartridge! That you have to blow on its end to clear any dust and it should do a satisfying click when it slots in.

Cooling might be an issue though...

This is part of the reason I wanted to announce it even before I feel ready to ship... The frontier models have such crazy velocity (admirable) and moving so fast that I didn't want them to make an announcement and suddenly trivialize all the work I 've put into this, overnight.

The fun technical challenges (that can also act as any sort of weak moat) are being taken away one by one, on an almost weekly cadence now! :)

So, this is what I have been thinking (and part of the reason why I want to "cook" it more before shipping). Since I have been building it for so long, I am not sure how much "overfit" it is on my data, the way I speak and the phrases I 'm using.

And of course the opposite too, how much the dog trained me to speak to it a certain way to maximize outcome success.

But then I thought, when people play games they are not using highly sophisticated vocabulary and there is probably lots of repetition since they are always under some form of multi-tasking stress (playing and replying/speaking). So maybe... maybe, the system can adjust itself. Use a big LLM offline to say "user said X, we did Y - was that good?" - then retrain itself.

The decomposer is basically a bunch of old-school embeddings/classifiers stitched together, it can train super fast and doesn't need tons of data. Could the thing calibrate itself to the user? Does it even need to? (because as I said I 'm a datapoint of 1 and I am not ready for the potentially huge stream of bug reports when I ship (add some perfectionism to the mix and you get the idea)).

edit: typos

Yes, the dog has a few levers including a "boredom" counter that will trigger it to do things on its own (like roam around, eat something, chase something (eg a rabbit) if there's one around etc). All connected with its current state (Eg hungry = will probably prioritize eating but not guaranteed). Other things are mapped to game events. Deterministic in nature but its underlying values can evolve over time (both its default "homeostasis value" but also the curve of which it increases). You can think of it as letting an llm design a behavior graph, and then playing that behavior graph until it changes again.

Talking to it alleviates pressure so that mechanism doesn't fire much, but in the void-mode video on the website, at the very end, it chooses to chase down an elk on its own much to my surprise.

After play testing it I had to tone it down quite a bit as it caused continuity issues...

In one scenario the dog was hungry so he went and ate something that was owned (counts as a crime attributed to the player) so we ended up in jail. Being transported to jail fired a new location event and a switch from "exterior->interior" which the dog responded to by having the llm author something along the lines of "It's nice to be finally indoors, it was getting cold outside". Which made for a very funny moment but a frustrating gaming experience.

At that point it's deterministic and doesn't need the LLM. Once a plan has been decomposed and evaluated, it becomes a sequence of actions. Actions can either wait on a trigger or flow from one to another.

For the arrow example: wait here -> bind: arrow event -> pick_up $target -> goto player -> drop $target.

A plan can bind to multiple steps and actions. eg you can tell the dog, "when I say apple you say banana" this creates a short lived "player has said %apple%" -> say "banana" rule. Hide and seek does "say: count to 10" (this then gets fed to llm that converts it to "one, two, three"). So, the plan decomposition is fuzzy, but it gets decomposed to concrete steps, these steps might invoke the LLM back if needed so, but mostly for color, not logic.

Where the LLM is used afterwards, is for evaluating the result. The dog might ask for a treat if it got right or get frustrated is it misses a beat, etc (depending on emotional state, hunger levels, etc).

Basically trying to use the best of each system (fuzzy/vague/emotional for LLM, "hard" game actions for determinism). You can think of Ale's output as a small flowchart.

I am using a pretty dumb model. It's great for speaking, terrible for thinking, so there's a need for creativity

I will be open sourcing it soon :) my code is a bit dirty (the whole system is 3 pieces. The game adapter / a websocket bridge between game and brain / the brain itself) and it runs on 2 different (local) machines currently.

Ale is what makes this work locally, I felt a little conscious about it as I am not sure if it is a novel approach or somebody comes out and claims I rediscovered BERT or something (though ale runs at 1/10 the cost of BERT).

Hello, author here. I was intentionally a little vague about this because this is kind of the thing that makes the whole approach work.

An LLM predicts the next token. If you're trying to predict the next token in a mathematics competition, or while playing a deep strategy game, being a much larger and more capable model helps enormously. To predict that next token correctly, the model effectively needs to model a bunch of possible future states - even if that is a second order (unintended) effect, it is what is seems to be happening.

This is basically the Ilya (and Dario) argument that prediction, understanding, and compression are the same thing (deep rabbit hole) from a few years ago.

----

In my opinion; this is a beautiful idea, but videogames do not need most of that. Videogames (and games in general) shine when character behavior is predictable, and when NPCs are a little dumb (just a little).

We already have very good small roleplaying models — Qwen 3.5 4B/9B/30B-A3B. Nowhere near frontier models at general reasoning. But they can act and write in a very engaging way. Good at roleplaying but very weak at reasoning. They just need a little nudge at reasoning...

And that's the key. The player has already expressed their intent: attack that guy, go look over there, cover me, find the key that shines and is golden, etc. A constrained world, with a constrained set of actions. Instead of asking the model to reason over an enormous space of possible futures, we're mostly asking it to map: player intent + current world state → a small sequence of plausible actions.

As for the "dump context to an LLM". It's basically. "You are roleplaying as X - you experienced Y - you like/dislike (dispositions) Z, you remember Alpha, your journal says Delta. Player orders you to do Gamma. - "What do you respond and do?"

It kind of works (as you can see in the videos I posted). I am not going against the grain, big models are better, but do we need those models for everything?

While I agree with the premise, I think this angle only applies on work one was going to do no matter what. The real power of these tools is that there are so many ideas people would like to try, but never have the time or motivation to pursue.

So the comparison is not only "built with and without LLM" but "would you even build this if you didn't have the LLM?". The gap in productivity in this case is much more wide.

Really fun project, but doesn't take ageism into account, it gets easier the further you get whereas it should get be getting harder in certain ways

There are also solutions for sharing your homelab with others (basically tunneling from your machine->server (internet accessible) <-> client. Though, if your machine would go to sleep that whole chain would fall apart. A few good automatic solutions out there that solve the problem (no "just replace dropbox with ftp" type of argument).

However, I see the appeal of this. Kind of surprised it hasn't happened yet to be honest.

Yes, you can see an early demo video in the page, of a very early version (I decided not to post the final wip yet).

I’m building an AI gaming companion. Essentially an AI NPC (a speaking dog, think dogmeat of fallout on steroids) that is grounded in the game world, can perceive what’s happening, perform actions, follow complex multi-step instructions, and strategize with the player. I want it to feel natural, so I 'm aiming for real audio conversationality (no push to talk & wait), so audio is important, but also... every millisecond trimmed from audio can be dedicated to heavier AI systems (caching helps heavily there, but extracting player's intent before it even reaches an LLM is probably the biggest trick).

Budgets are pretty tight, ~400ms for everything. I have optimized qwen-3-asr and added streaming detection to it, as well as pocket-tts (this) and qwen-3-tts, with a goal of ~70ms for audio for both detection/creation and then ~350ms for the planning.

It's interesting that gpt-live was just announced today, planning to make more things open source as well as the learnings while building this.

Hi HN — I’ve been working on a real-time conversational AI, and one of the biggest problems is speech latency. As I was experimenting with various TTS systems I stumbled upon Pocket-TTS by Kyutai. It runs on the CPU which made it a great candidate since it keeps speech synthesis away from GPU resources.

As I was using it I noticed that it didn't solve the latency issues fully due to leading silence and artifacts generation (quite common with all these systems). So I started poking under the hood with the help of agents to try and improve it.

On m4 max, the new browser build runs around ~14x realtime on longer utterances, and around ~3.5x realtime on an iPhone 16 Pro. The native CPU path is faster still: roughly 32–33x realtime warm, with first useful audio around 30ms on my benchmarks. Other systems will vary; my Windows browser results were slower but still usable. This does not introduce a new model or claim better voice quality. The goal was to preserve the intended PocketTTS quality while making it fast enough for low-latency interactive scenarios.

I decided to open source it in case it's useful for anyone else, or at least as a demo of what things that are finally possible now that the barrier of trying stuff has been lowered.

EDIT: Keep in mind it downloads 67mb of data on load.

And I think both have seen these much older handcrafted css3d engines

- https://pantel.is/projects/css3d/

- https://keithclark.co.uk/labs/css-fps/ (the original)

but quake and doom took it to the next level :)