The Mobile AI Frontier: Architecting for Zero-Egress Intelligence

On-device AI inference on mobile devices operates under a constraint that datacenter GPUs never face: the battery. A single inference pass on the Apple A19 Neural Engine consumes 50-200 mJ depending on model size and sequence length. For a conversational AI agent processing 100 requests per hour, the daily inference energy consumption reaches 120-480 J (0.033-0.133 Wh) — a small fraction of a 4,000 mAh (14.8 Wh) phone battery. However, continuous background inference — such as an ambient voice assistant or an always-on camera — can drain the battery in under 4 hours if not carefully scheduled.

Battery-aware scheduling uses the device's **Energy Model** — a per-NPU power curve that maps inference workload parameters (model size, batch size, sequence length, NPU frequency) to energy consumption. The model is calibrated at manufacturing time and stored in the device's power management firmware. When an application requests inference, the **Energy-Aware Scheduler (EAS)** in the kernel estimates the energy cost and checks the current battery level. If the battery is above 50%, all requests are serviced immediately at maximum NPU frequency (1.5 GHz on A19). If the battery is below 20%, the scheduler caps the NPU frequency to 800 MHz, reducing energy per inference by 40% at the cost of 2x latency.

The scheduler also considers **Thermal Headroom** — the temperature margin before the device's skin temperature exceeds 45°C (the regulatory limit for handheld use). At 800 MHz, the NPU generates 2.5W of heat compared to 5.2W at 1.5 GHz. The thermal time constant of a smartphone chassis (approximately 200 seconds) means the scheduler can burst at 1.5 GHz for up to 30 seconds without exceeding the thermal limit, provided it then idles for 170 seconds. This **Burst-and-Cool** cycle allows the user to experience low-latency inference for short interactions while protecting both battery life and thermal comfort.

The most sophisticated battery-aware schedulers incorporate **Prediction Models** that learn the user's inference usage pattern. If the user typically runs a 5-minute language translation session every morning at 8 AM, the scheduler pre-warms the NPU by loading the model into SRAM and setting the frequency to 1.5 GHz at 7:58 AM, anticipating the request. This predictive pre-warming reduces cold-start latency from 800 ms to under 50 ms while using the same energy budget because the NPU would have been powered up anyway at the scheduled inference time. Google's Tensor G6 implements this through the **Adaptive Inference Predictor (AIP)** — a lightweight on-device model that achieves 85% accuracy in predicting the next inference request within a 30-second window.

Quantization is the lever that makes a 7B model fit in a phone, but it is not free. Moving from FP16 to INT8 halves weight memory; moving to INT4 quarters it. The range and resolution math governs the quality cost. A symmetric INT8 representation maps the FP16 range [-127, 127] linearly, giving a step size of scale = max|w| / 127 and a relative quantization error bounded by 0.5 x scale / |w| — roughly 0.4% for uniformly distributed weights. INT4 is harsher: symmetric INT4 uses only the range [-8, 7], so the scale is twice as coarse per bit, and the 0.5-step quantization error lands near 6% for weights near the midpoint. In practice INT4 weights alone are tolerable, but the same INT4 applied to activations destroys output quality, which is why production mobile stacks quantize weights to INT4 and leave activations in INT8 or FP8.

The dominant quality killer is not the average weight error but the activation outliers — the handful of activation channels whose values are 20-100x larger than the median. If the INT8 scale is set from the max channel, the remaining 99% of channels quantize to a handful of integer levels and the model's representational resolution collapses. Two families of fixes exist. Per-channel (or per-group) quantization gives each output channel its own scale, which collapses the dynamic range problem for weights; for activations, the same trick works but costs extra hardware support, which many NPUs lack. The cheaper fix is activation scaling, the idea behind SmoothQuant: a mathematically equivalent rescaling of weights and activations that moves the outlier amplitude into the weight matrix (which is quantized per-channel) and leaves activations with a gentle, hardware-friendly distribution.

The second frontier is the KV-cache, which compounds the pressure because it grows with sequence length. Quantizing the cache to INT8 or INT4 is now standard, but the right choice depends on where in the conversation the cache is read. Early tokens are read more often and their errors propagate through every subsequent token, so mixed-precision schemes store the first 10% of KV entries in INT8 and the remainder in INT4. Measured across a 4,096-token session, that single optimization recovers 55% of the cache memory of a full-INT8 design while keeping perplexity degradation under 0.3% — a trade the NPU performance benchmarks capture as end-to-end tokens-per-watt rather than raw TOPS.

Finally, hardware matters more than software here. Qualcomm's Hexagon executes native INT4 matrix operations in its tensor unit, which means an INT4-quantized model runs at nearly the same throughput as INT8; Apple's ANE, by contrast, historically emulated INT4 through INT8 primitives, halving the expected speedup. This is the reason the 2026 guidance is "quantize to what the silicon does natively": the modeler should evaluate the effective TOPS at each precision before committing the graph, then pick the precision that maximizes throughput within the thermal and battery budget — the same optimization exercised by GPU performance modeling tools at cluster scale, but with a millijoule-per-token budget instead of a megawatt one.

Attention architecture is the second memory lever, orthogonal to quantization. Multi-Head Attention (MHA) stores a distinct Key and Value tensor for every head; Multi-Query Attention (MQA) shares one K/V set across all heads; Grouped Query Attention (GQA) shares K/V within groups. The KV-cache size is 2 x L x G x H_kv x S bytes, where L is layers, G is the number of K/V groups, H_kv is the per-group head dimension, and S the sequence length. A 28-layer, 7B model with H_kv = 128 and G = 8 (that is, 8 K/V groups shared by 32 query heads) produces 2 x 28 x 8 x 128 x 4096 = 235 MB of KV cache at FP16 for a 4,096-token context. The same model in full MHA (32 K/V heads) would need 4x more — nearly 1 GB — which is why every mobile LLM released in 2025 and 2026 ships GQA-8 or GQA-4 rather than MHA.

Combining GQA with INT4 KV quantization drops the 235 MB to 59 MB, and that is the number that makes long on-device conversations viable: the weights of a 7B INT4 model occupy roughly 3.9 GB, leaving several hundred megabytes of the 8 GB unified pool for the cache and the OS. The remaining lever is eviction. Sliding-window attention keeps only the most recent W tokens (typically 2,048) fully materialized, and re-computes or fetches earlier context from flash on demand — trading a modest latency spike for a bounded cache. For conversational agents, a hybrid policy performs best: the system prompt and the first user turn are pinned in a permanent cache region (they are re-read on every request), the most recent 2,048 tokens occupy the sliding window, and everything between is compressed to INT4 or evicted.

Paged attention, borrowed from datacenter serving, applies at the device level too: KV entries are stored in fixed-size pages that can be scattered across the memory map, so the allocator does not need one contiguous block per conversation. On an 8 GB device with a fragmented heap after an hour of app usage, contiguous allocation fails and the whole assistant would otherwise be unloaded; paged allocation succeeds because it needs only 512-byte pages. Measured on the Qualcomm Hexagon, paged attention with 512-byte pages reduces cache allocation failures from 18% to under 0.1% in a 32k-context scenario, with no measurable latency penalty because the NPU's address translation unit handles the indirection in hardware.

The final interaction is with the memory bus, not the compute. Every token generated reads the full KV cache once; at 235 MB of FP16 cache and 60 tokens per second, the NPU must sustain 235 MB x 60 = 14.1 GB/s just for cache reads, a quarter of a 150 GB/s unified-memory budget. Slashing the cache with GQA-8, INT4, and sliding windows is what brings that traffic down to a few GB/s and keeps the phone cool enough to hold. This memory-traffic view of inference — not FLOPs — is the correct frame for edge AI inference on ARM and RISC-V devices, and it is the same reasoning that motivates KV-cache offload to fast storage, the technique the DPU offload literature applies to cluster interconnects.