Google Sped Up Gemini Nano on Pixel Without Retraining Its Backbone

Google sped up Gemini Nano on Pixel phones without retraining its backbone. Google says notification summaries and proofreading became significantly faster; on Pixel 9 devices, task-dependent generation speedups reached 50% or more. The system still accepts only tokens the original model would have chosen.

Gemini Nano runs on the phone instead of sending each generation job to a remote model. This on-device inference keeps the work close to the product, but forces the model to operate within a phone’s tighter memory, computing, and energy limits.

LLM progress is usually measured by the capabilities of a new model. Google’s work points to a separate track: keep the core model and improve the machinery that runs it, allowing teams to manage capability development and execution engineering as distinct workstreams.

How Google made generation faster

Language models produce text in pieces called tokens. A token may be a word, part of a word, or punctuation. Most models generate one token, add it to the output text, then run again to choose the next token. This repeated process is called autoregressive generation.

Google’s approach was to reduce the number of passes needed to produce output.

A transformer is the architecture behind many language models. At a high level, it consists of stacked layers that turn the input into representations used to predict the next token. Those layers make up much of the model’s backbone. The backbone’s learned parameters, or weights, carry much of the model’s language capability and behaviour.

Google kept the Gemini Nano v3 backbone’s weights frozen and trained a small component attached near its output. This component uses multi-token prediction, or MTP, to propose several future tokens for the backbone to check instead of just one.

Imagine an assistant drafting the next few words from notes the lead writer has already assembled. The lead writer accepts only the words they would have chosen.

As the backbone processes text, it produces a hidden state, a numerical representation of the context built so far. It also maintains a key-value cache, which stores attention information from earlier tokens.

Traditional speculative decoding often uses a second, smaller language model as the drafter. The smaller model processes the prompt, maintains its own cache, and proposes tokens for the larger model to verify.

Google attached the drafting head to the existing Gemini Nano backbone. It takes the backbone’s final hidden states as input and uses cross-attention to read the existing key-value cache. The head therefore avoids processing the prompt again or maintaining a second dynamic cache, reducing prefill work and runtime memory use—resources in short supply on mobile devices.

Google removed the second prompt-processing path and dynamic cache used by a standalone drafter; its attached head reuses the backbone’s internal state, while the backbone still determines which tokens are accepted.

The verifier accepts a proposed token only when it matches the backbone’s choice. Correct proposals let the system accept several tokens in one pass. A mismatch ends the accepted sequence.

Under this strict verification method, Google reports bit-for-bit identical accepted output.

An on-device system has at least four layers worth separating:

1. The backbone carries the core capability.
2. An attached component, such as a drafting head, can help the backbone generate more efficiently.
3. The runtime and hardware execute both parts within the device’s limits.
4. The product workload determines what the system must do.

On Android, AICore handles model distribution, hardware acceleration, weights, adapters, and safety features. ML Kit exposes product-facing APIs. Developers remain responsible for client and application safety.

Calling all four layers “the model” hides the source of an improvement. Faster output might come from a new backbone, a better decoding method, tighter runtime integration, different hardware, or a workload that suits the method.

The Pixel result changed the attached decoding component while keeping the backbone’s learned weights frozen. Separating the four layers makes the source of an improvement visible.

The evidence card

Google reports workload-dependent generation speedups of 50% or more, depending on the task. The result text names Pixel 9 devices; the comparison footnote says the pre-update baseline covers Pixel 9 and Pixel 10 phones.

Google also reports up to a 55% improvement in token acceptance for predictable text structures. The sentence containing that figure does not state its comparison baseline.

The attached design saved 130 MB of memory per instance compared with a standalone drafter.

In production notification summaries and proofreading, Google reports nearly two additional accepted tokens per inference pass on average. Accepting more tokens requires fewer verification passes. Google attributes that reduction to lower processor use and better energy efficiency.

What came before

Multi-token prediction predates the Pixel implementation.

In 2024, Fabian Gloeckle and his co-authors trained transformers with several future-token prediction heads. These heads can support exact self-speculative decoding, in which the target model helps draft its own future tokens. A later Llama 2 continuation added four-token prediction while training on another 200 billion code tokens but found no significant overall improvement. The authors suggested that changing the training objective may have disrupted the pretrained model.

Also in 2024, the EAGLE paper described a related route: keep the target model fixed and train a small auxiliary decoder to draft from the target model’s internal features. EAGLE reported speedups on server GPUs, not phones.

Samsung researchers supplied the strongest independent mobile comparison in the reviewed sources. In a 2026 experiment, they used a frozen inference graph, runtime LoRA adapters, and self-speculative decoding with a multilingual LLaMA-based model on Galaxy S24 and S25 phones. They reported up to 2.3 times faster decoding, but their architecture differed from Google’s attached-head design.

A second track for edge-model development

The sources justify three levels of confidence.

Demonstrated: Google reports a Pixel production retrofit that adds an attached head to a frozen backbone rather than retraining Gemini Nano. Samsung researchers report another mobile design that combines a frozen inference graph with runtime adaptation and speculative decoding.

Credible signal: Teams can work separately on capability, runtime, task adaptation, and decoding. The Google and Samsung examples use different architectures, but both separate some execution work from backbone development.

Unresolved: The published evidence does not establish how widely these methods transfer across models, devices, workloads, or release processes. Absolute energy use, thermal effects, update cadence, total development cost, validation savings, and production adoption beyond Google remain unanswered.

More work stays on the device

As on-device models improve, product teams can keep frequent, predictable, latency-sensitive work on the device, using cloud models for requests that need greater capability, broader context, or more compute. Android already presents on-device Nano and cloud Gemini as complementary options. Apple processes requests on-device when possible and uses Private Cloud Compute when greater computational capacity is needed.

A hybrid mobile architecture can keep suitable work on the device and reserve cloud inference for requests whose capability, context, or compute requirements justify a remote call.

For regulated industries, that creates a practical opening: teams can ship bounded features while they work through the privacy, security, and approval demands of cloud inference. When a task can run fully on-device, local processing can keep sensitive data from being sent to a remote service, remove the network connection from the inference path, and avoid the per-call cost of remote inference. The European Data Protection Supervisor notes that local processing can reduce latency, bandwidth use, and data-transmission costs.

Local processing does not make a product compliant or secure by itself. Telemetry, model updates, access control, compromised devices, auditability, and output safety remain product concerns. Local inference narrows one part of the risk surface; it does not erase the rest.

The more consequential shift is in the default architecture. A mobile application will not need the best cloud model for every step. It can keep suitable work local and escalate only when the additional capability, context, or compute justifies the cost, latency, and data exposure of a remote call. Cloud inference becomes an escalation path that must earn its use.

Sources

• Google Research, “Accelerating Gemini Nano models on Pixel with frozen Multi-Token Prediction,” June 26, 2026.
• Fabian Gloeckle et al., “Better & Faster Large Language Models via Multi-token Prediction,” 2024.
• Yuhui Li et al., “EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty,” revised 2025.
• Sravanth Kodavanti et al., “Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM,” 2026.
• Android Developers, “Gemini Nano,” updated April 2, 2026.
• Android Developers, “Find the right AI/ML solution for your app,” retrieved July 19, 2026.
• Apple, “Apple extends its privacy leadership with new updates across its platforms,” June 10, 2024.
• European Data Protection Supervisor, “On-device artificial intelligence,” retrieved July 19, 2026.