nmitchko

The goal of this auxiliary module is to have the model recurse pre-emit trained on good thinking traces. This includes a 6-to-1 compression of thinking tokens.

Therefore output tokens are decodeable, but are trained compressed. So they are approximations of faster thinking.

Interpretability is a mixed bag even with trained tools on top of existing models.

Fair criticism, i’ll rewrite it… as this is a pet project and post based on another HN comment.

After vLLM startup I observed this on first request only then all further conversations were fine, new and original. Can’t figure out why but seems to be fine after that.

My main motivation is to be able to tune the thinking effort without polluting token IO. Inspiration came from the CoLaR paper and other works showing that repeating laters in an LLM got better results.

I am on my own GPUs so thought it be cool to reduce token output. you can charge per latent loop as a token i guess.

I wanted a knob to keep thinking going until it was sure it was done.

I can do that, main difference seems to be this ablation seems to over think simple prompts, but can do a proper eval (might take 2-3 days)

Fair critique, ill rewrite (this is just my personal pet project but thought was cool to share here)

It’s an adaptation of CoLaR, but my implementation is a little different: - Dedicated stop head to fire when latent thinking hits threshold - MTP support with training taking draft support as first class - different architectural layer 35 -> layer 42 writeback. So latents skip roughly 6.2 tokens of reasoning per token, then never make it to decoded output

Perhaps it might be interesting: a latent thinking version is here https://huggingface.co/nmitchko/DeepSeek-V4-Flash-0731-Laten...

Does no thinking emissions for context saving.

Author of this model - I was curious how large models do adaptive thinking and it got me down the path of implementing a model with latent only thinking.

The part I think is most interesting is that the reasoning never touches your context window. The model thinks in a compressed 1024-dim latent space, then decodes just the answer, so a ~6-token reasoning step collapses into one latent step. It also self-terminates: a learned stop head ends the thinking when it's done, so easy questions budget a few steps and hard ones run up to ~256.

The tradeoffs: Blackwell-only (NVFP4), on a vLLM fork (upstream can't serve it due to hash-MoE routing), the trace isn't interpretable, and eval is BBH-only. Happy to answer questions about the closed-loop runtime:: getting the latent injection onto the cudagraph fast path was the fiddly part.

Poster: The cool part is that the model literally talks itself out of giving the cached wrong answer with a latent only thinking budget.

A fantastically simple solution to improving algorithms, I wish I had this years ago in activation engineering: https://blog.n.ichol.ai/llm-activation-engineering-an-easy-f...

How do I access AlphaEvolve?

I used those two in combination to fix pain after 3x surgeries to repair a torn pec + infection. They work and helped me heal from being at a 3/10 constant pain down to baseline.

Not something I would do at any point for fun. But anecdotally, it's materially better than other alternatives offered/available.

Can someone make a startup that allows me to do this as an individual?

In case anyone wants to do this themselves, check out the pipeline here: https://github.com/isc-nmitchko/iris-document-search

Colnomic and nvidia models are great for embedding images and MUVERA can transform those to 1D vectors.

Next steps for AI in general:

  - additional modalities
  - Faster FPS (inferences per second)
  - Reaction time tuning (latency vs quality tradeoff) for visual and audio inputs/outputs
  - built-in planning modules in the architecture (think premotor frontal lobe)
  - time awareness during inference (towards an always inferring / always learning architecture)

No other models that are public worth comparing to... Hippocratic advertises good benchmarks but that might be marketing fluff.

Have you checked out dataset building with nemotron? The nemotron synthetic data builder is quite powerful.

Moreso, check out model merging. It's possible if you merge some of your model against llama3.1 base it may perform much better.

Check out max labonne's work on hugging face

Interesting they don't compare to open-bio. Page 7 charts are quite weak.

https://huggingface.co/aaditya/Llama3-OpenBioLLM-70B

We're excited to share pitchpilot with the HN community. Our beta users have found the embedded audio particularly useful for enterprise sharing. We're keen to keep improving, and our mission is to make communication easier.

In the roadmap is adding video export, digital twin presentations, and real-time presentations. We don't wrap a public LLM, so we don't share any data.

Given that Generative AI can now read brain scans [1] and this, I wonder how far away we are from "you thought negatively about something, the authorities are on their way".

[1] -- https://www.biorxiv.org/content/10.1101/2022.11.18.517004v3

Tin-foil hat time:

1. First, models will predict pollution. The outcomes will help shape urban policy. But these won't solve crime or stop people from driving.

2. Second, models will predict individual behavior and track person level emissions. The outcomes will force behavior changes, mostly freedom limiting.

3. Third, and finally, models will predict thoughts. The the thought of driving instead of walking might trigger a response.

It's a slippery slope and we need to draw a line between prediction and policy.

How does this compare to ehealthexchange or other qhins that have many years of experience and charge lower costs?

It truly feels like the space race in terms of building LLMs right now. Question is, who lands on the moon first?

This reminds me of the matrix movie scene when they look at the encrypted thoughts of the matrix.

https://cdn.swisscows.com/image?url=https%3A%2F%2Fi.pinimg.c...

Great work, will try this tonight.

Only question, why do you name variables with the λ symbol?