fzysingularity

It doesn’t seem like they have monopoly here given how big Uber Eats is?

Yeah, it’s surprising that this is the runtime model they went with as opposed to a hybrid edge/cloud model. I’d love to see a system 1 on edge, system 2 on cloud offering. They’re likely the best positioned to do this.

VLM Run (https://vlm.run) | 1x Founding Infrastructure Engineer | In-Person (Bay Area) | Full-time

We’re building the inference platform for visual intelligence. We’re a deeply technical team of AI / computer-vision engineers (20+ years combined, MIT/CMU/NC State PhDs) who’ve shipped production ML infra across autonomous driving and LLMs.

We launched the VLM Run Gateway (https://vlm.run/gateway) in September, a unified API that serves open-weight VLMs, embodied VLAs, ViTs, served across a fleet of GPUs and clouds. That's exactly the infrastructure problem this role will own. We're already scaling to serve 100s of thousands of requests per day, so if this sounds exciting to you, read on.

Email us at [email protected] with your GitHub profile, papers, ML projects you’ve recently shipped (100+ GH stars only) - especially with Ray, k8s, GPUs, serverless. No AI text please, keep it short, shorter emails are more likely to get a response. No remote work, must be in the Bay Area (specify in email subject).

When you say Jev is optimized for it, I get that there’s no need for unnecessary decodes in an autoregressive fashion.

But both LLMs and Jev-like models would need to prefill, the only optimization Jev does differently is the decode which can be emulated by reading off logprobs.

We don’t know the param size of Jev, to determine the most comparable model, but if I had to guess it’s sub-100B.

I don’t know of one definitive resource but “constrained decoding”, regex/grammar-based decoding, json-schema decoding will give you a bunch of hits. Look up vLLM, SGLang and outlines’ implementations for more technical details.

This looks pretty decent: https://www.aidancooper.co.uk/constrained-decoding/

FWIW you’re absolutely correct on how most developers are unaware of this as they’re mostly operating on the API level and not aware of server side (vLLM/SGLang) capabilities.

The aspect that I like the most is the typesafe API that introduces new probabilistic concepts that are more sound than json schema and constrained decoding with quasi-confidence scores. Developers were asking LLMs to also emit confidences which made absolutely no sense whatsoever.

Mostly just 3d world models though, they're not the same as multimodal models.

We just did exactly this - added TypeSafe-compatible API (incl. websocket support) for various VLMs. Latency right now is <250ms, but will be able to get it to <150ms (p95).

Take a look at a snake demo with streaming image inputs: https://x.com/spillai/status/2103630735425089957

Is the LiDAR sufficiently accurate at those distances?

Curious about this: do you expect that Claude science will also scoop interesting directions looking at anonymous usage, to publish posts like this before the original author (similar to the Navier Stokes debacle)?

Am I missing something here:

p(y = next thinking+decision token | x = question) != p(y = next decision token | x = question)

The former is what LLMs are trained for, the latter is what Jev was likely trained on (likely used thinking alignment as an auxiliary loss, but not explicitly included in the probability calibration).

Very cool to see the openness here, and likely more like this will come from smaller startups where they win users on transparency.

text is mostly beaten to death, so you can expect good defaults to work for vllm. VLMs specifically are quite sensitive to quantization, especially if you want it to do fine-grained localization (time or spatial), and the vllm default params can be way off for your use-case.

For example, most vision models don't need 256K context length for modes like Qwen3.8-27B when all you care about is single-image captioning, so you can technically save on KV cache. Video reasoning does require that context length, so it's a different set of deployment parameters that need to be enabled.

All of this to say that the providers that offer these models, are simply using vLLM / SGLang, and mostly cater to the text inference use-case (coding, etc). Vision always seems to be a bit of an afterthought.

The author’s comments on vision providers is especially interesting. We saw that most providers don’t provide native video url support, have high-variability in vision performance (likely due to the fact that they’re serving different quantization levels behind the same model id).

If you’re building vision-native apps, there are so many footguns in vLLM/SGLang serving configurations, let alone the routing/orchestration in providers like OR, that leave the user more confused about the model’s capabilities.

Have you tried any vision tasks with this model? We've been serving these on our gateway [1], and the quants are quite terrible for vision. Curious to hear your experience.

[1] https://www.vlm.run/gateway

I'm surprised this post isn't getting as much attention as it should. Crazy times!

VLM Run (https://vlm.run) | 1x Founding Infrastructure Engineer

We’re building the inference platform for visual intelligence. We’re a deeply technical team of veteran AI / computer-vision engineers (20+ years combined, MIT/CMU PhDs) who’ve shipped production ML infrastructure across autonomous driving and LLMs.

We just launched the VLM Run Gateway (https://vlm.run/gateway), a unified API that serves open-weight VLMs, embodied VLAs, ViTs, served across a fleet of GPUs and clouds. That's exactly the infrastructure problem this role will own.

If you’re interested, email us at [email protected] with your GitHub profile and link to projects you’ve recently built - especially with docker, k8s, GPUs.

I think we all ought to look at the ZDR fine-print here.

I get that in principle that there's no retention, but these are powerful models that can comprehend, paraphrase and summarize your logs for the sake of "product" improvement. Who knows what's collected here.

The ICP question was more around the model itself. Are they looking to license it to robotics companies? Do they imagine that devs at robotics companies would be willing to deploy these models as a black box?

It’s unclear to me what their desired outcome for a blog post like this. If you’ve ever worked in a robotics setting, 80% implies that 20% of your autonomous actions are incorrect. Imagine if this were the case for autonomous driving where your car misbehaves 1 in every 5 actions it takes.

Posts like this just reminds me of the end to end demos AV companies built in the early days using a single camera - only to realize that it’s harder than it looks years later into development.

Frontier labs are realizing that software/models themselves don’t have real moats and move to embodied ai.

SOTA 80% means a practically useless robot. What are they really imagining their ICP to be here?

It's live now, https://github.com/vlm-run/mm.

We were planning to open-source this soon, but jumped the gun and posted about the video encoders here since it seemed relevant.

In either case, here you go, it's public now: https://github.com/vlm-run/mm.

Exactly! We experimented with a whole bunch of video encoding techniques for LLMs here: https://vlm-run.github.io/mm/encoders/#video

Pretty terribly expensive way to watch a video with Claude.

Use Gemini or some local VLM to do this way more efficiently. We spent quite a bit of time on video understanding, and Claude will just burn tokens.

Check out this library: https://vlm-run.github.io/mm/

You can swap models and try out different encoding methods for videos (https://vlm-run.github.io/mm/encoders/#video)

This is neat. I'd love to figure out a sequence of emojis that triggers the LLM in ways that puzzles a human.