News for hackers. Stuff that matters.
Re:Introducing Gemma 3n by refulgentis on Thursday June 26, 2025 @02:08PM (#44389793)
Somethings really screwy with on-device models from Google, I can't put my finger on what, and I think being ex-Google is screwing with my ability to evaluate.

Cherry-picking something that's quick to evaluate:

"High throughput: Processes up to 60 frames per second on a Google Pixel, enabling real-time, on-device video analysis and interactive experiences."

You can download an APK from the official Google project for this, linked from the blogpost: https://github.com/google-ai-edge/gallery?tab=readme-ov-file...

If I download it, run it on Pixel Fold, actual 2B model which is half the size of the ones the 60 fps claim is made for, it takes 6.2-7.5 seconds to begin responding (3 samples, 3 diff photos). Generation speed is shown at 4-5 tokens per second, slightly slower than what llama.cpp does on my phone. (I maintain an AI app that inter alia, wraps llama.cpp on all platforms)

So, *0.16* frames a second, not 60 fps.

The blog post is so jammed up with so many claims re: this is special for on-device and performance that just...seemingly aren't true. At all.

- Are they missing a demo APK?

- Was there some massive TPU leap since the Pixel Fold release?

- Is there a lot of BS in there that they're pretty sure won't be called out in a systematic way, given the amount of effort it takes to get this inferencing?

- I used to work on Pixel, and I remember thinking that it seemed like there weren't actually public APIs for the TPU. Is that what's going on?

In any case, either:

A) I'm missing something, big or

B) they are lying, repeatedly, big time, in a way that would be shown near-immediately when you actually tried building on it because it "enables real-time, on-device video analysis and interactive experiences."

Everything I've seen the last year or two indicates they are lying, big time, regularly.

But if that's the case:

- How are they getting away with it, over this length of time?

- How come I never see anyone else mention these gaps?

Re:Introducing Gemma 3n by mlsu on Thursday June 26, 2025 @02:56PM (#44390239)
It looks to me by the marketing copy that the vision encoder can run 60FPS.

> MobileNet-V5-300M

Which makes sense as it's 300M in size and probably far less complex, not a multi billions of parameters transformer.

Re:Introducing Gemma 3n by refulgentis on Thursday June 26, 2025 @03:03PM (#44390303)
I agree that's the most likely interpretation - does it read as a shell game to you? Like, it can do that but once you get the thing that can use the output involved it's 1/100th of that? Do they have anything that does stuff with the outputs from just MobileNet? If they don't, how are they sure I can build 60 fps realtime audiovisual experiences they say I can?
Re:Introducing Gemma 3n by namibj on Thursday June 26, 2025 @05:44PM (#44391791)
Classify/similarity/clustering works fine with just an encoder, doesn't it?

I guess there's benefit to running that step without subsampling to the initial 256 tokens per image/frame ( https://ai.google.dev/gemma/docs/gemma-3n/model_card#inputs_... ) to go on from that, https://github.com/antimatter15/reverse-engineering-gemma-3n suggests these are 2048 dimensional tokens, which makes these 60 Hz frame digestion rate produce just under 31.5 Million floats-of-your-choosen-precision per second. At least at the high (768x768) input resolution, this is a bit less than one float per pixel.

I guess maybe with very heavy quantizing to like 4 bit that could beat sufficiently-artifact-free video coding for then streaming the tokenized vision to a (potentially cloud) system that can keep up with the 15360 token/s at (streaming) prefill stage?

Or I could imagine just local on-device visual semantic search by expanding the search query into a bunch of tokens that have some signed desire/want-ness each and where the search tokens get attended to the frame's encoded tokens, activation function'd, scaled (to positive/negative) by the search token's desire score, and then just summed over each frame to get a frame score which can be used for ranking and other such search-related tasks.

(For that last thought, I asked Gemini 2.5 Pro to calculate flops load, and it came out to 1.05 MFLOPS per frame per search token; Reddit suggests the current Pixel's TPU does around 50 TOPS, so if these reasonably match each terminology wise, assuming we're spending about 20% of it's compute on the search/match aspect, it comes out to an unreasonably (-seeming) about 190k tokens the search query could get expanded to. I interpret this result to imply that quality/accuracy issues in the searching/filtering mechanism would hit before encountering throughout issues in this.)

Re:Introducing Gemma 3n by refulgentis on Friday June 27, 2025 @05:59PM (#44400637)
There's a lot of Not Even Wrong, in the Pauli sense, going on presumably because back-of-napkin-with-LLM is like rocket fuel, I love it. :) But, the LLM got ahead of understanding the basics. I could write probably 900 words. Lets pull one thread out as an example:

> I guess maybe with very heavy quantizing to like 4 bit that could beat sufficiently-artifact-free video coding for then streaming the tokenized vision to a (potentially cloud) system that can keep up with the 15360 token/s at (streaming) prefill stage?

the 6-7s I am seeing is what it costs to run an image model, even running on GPU on M4 Max with 64GB of GPU RAM. This repros with my llama.cpp wrapper, and the llama.cpp demo of it.

It is simply getting tokens that is taking that long.

Given that reality, we can ignore it, of course. We could assume the image model does run on Pixel at 60 fps, and there's just no demo APK available, or just say it's all not noteworthy because as the Google employee points out, they can do it inside Google, and external hasn't been prioritized.

The problem is that the blog post is announcing this runs on device at up to 60 fps today, and announces $150K in prizes if you work based on this premise. We have 0 evidence of this externally, the most plausible demo of it released externally by Google is running at 1/500th of this speed, and 1 likely Google employee is saying "yup, it doesn't, we haven't prioritized external users!" The best steelman we can come up with is "well, if eventually the image model runs at 60 fps, we could stream it to an LLM in the cloud with about 4 seconds initiate + prefill latency!"

That's bad.

Re:Introducing Gemma 3n by catchmrbharath on Thursday June 26, 2025 @02:29PM (#44389989)
The APK that you linked, runs the inference on CPU and does not run it on Google Tensor.
Re:Introducing Gemma 3n by lostmsu on Thursday June 26, 2025 @02:51PM (#44390193)
How does their demo work then? It's been 3 months since 3n was first released publicly.
Re:Introducing Gemma 3n by refulgentis on Thursday June 26, 2025 @09:52PM (#44393127)
What demo?

The only one we have works as described, TL;Dr 0.1 fps.

Re:Introducing Gemma 3n by refulgentis on Thursday June 26, 2025 @02:38PM (#44390070)
That sounds fair, but opens up another N questions:

- Are there APK(s) that run on Tensor?

- Is it possible to run on Tensor if you're not Google?

- Is there anything at all from anyone I can download that'll run it on Tensor?

- If there isn't, why not? (i.e. this isn't the first on device model release by any stretch, so I can't give benefit of the doubt at this point)

Re:Introducing Gemma 3n by catchmrbharath on Thursday June 26, 2025 @04:37PM (#44391076)
> Are there APK(s) that run on Tensor?

No. AiCore service internally uses the inference on Tensor (http://go/android-dev/ai/gemini-nano)

> Is there anything at all from anyone I can download that'll run it on Tensor?

No.

> If there isn't, why not? (i.e. this isn't the first on device model release by any stretch, so I can't give benefit of the doubt at this point)

Mostly because 3P support has not been a engineering priority.

Re:Introducing Gemma 3n by refulgentis on Thursday June 26, 2025 @05:20PM (#44391533)
> Mostly because 3P support has not been a engineering priority.

Got it: assuming you're at Google, in eng. parlance, it's okay if it's not Prioritized™ but then product/marketing/whoever shouldn't be publishing posts around the premise it's running 60 fps multimodal experiences on device.

They're very, very, lucky that ratio of people vaguely interested in this, to people follow through on using it, is high, so comments like mine end up at -1.

Re:Introducing Gemma 3n by my123 on Thursday June 26, 2025 @07:30PM (#44392471)
Tensor is essentially shipping subpar hardware with not even taking care of software properly.

https://ai.google.dev/edge/litert/android/npu/overview has been identical for a year+ now.

In practice Qualcomm and MediaTek ship working NPU SDKs for third party developers, NNAPI doesn't count and is deprecated anyway.

Re:Introducing Gemma 3n by refulgentis on Thursday June 26, 2025 @09:43PM (#44393100)
Man this is a funny situation. Ty for sharing, more or less confirms my understanding. Couldn't quite believe it when I was in Google, or out of Google. This should be a big scandal afaict. What is going on???

(n.b. to readers, if you click through, the Google Pixel Tensor API is coming soon. So why in the world has Google been selling Tensor chips in Pixel as some big AI play since...idk, at least 2019?)

Re:Introducing Gemma 3n by my123 on Thursday June 26, 2025 @11:02PM (#44393367)
Yes, you can use first-party models on the Pixel NPUs or you're stuck with NNAPI which is self-admittedly deprecated by Google and doesn't work all that well.

On third party model workloads, this is what you will get:

https://ai-benchmark.com/ranking.html

https://browser.geekbench.com/ai-benchmarks (NPU tab, sort w/ quantisation and/or half precision)

Google is clearly not serious on Pixels in practice, and the GPU performance is also behind by quite a lot compared to flagships, which really doesn't help. CPUs are also behind by quite a lot too...