mattmireles/magenta-realtime-2-iphone

Magenta RealTime 2 on the iPhone, Without the GPU

Live music generation at 25 frames per second, on the phone in your pocket. Ten unbroken minutes of 48 kHz stereo without melting your phone. Zero dropouts. Even on a 2020 iPhone 12 Pro. And the GPU never wakes up.

This is Google DeepMind's Magenta RealTime 2 (mrt2_small, 230M parameters), surgically cut into three Core ML graphs and placed on the silicon each one wants. The temporal transformer holds p99 ≈ 14 ms on the iPhone's Neural Engine against a 40 ms frame budget. Output correlation vs Google's MLX reference: 0.999985904188 — we publish all twelve digits because we measured all twelve. Decoder SNR: 118.85 dB. Depth rollout tokens vs the reference sampler: 0/900 mismatched in FP32. App-attributed GPU time in a 60-second Instruments capture of the all-ANE pipeline: zero — the only process with GPU intervals is iOS's screen compositor.

Exporters, validation harness, full method docs: github.com/mattmireles/magenta-realtime-2-iphone Pre-converted models (this repo): you are here.

Corrected artifact generation (paper Surgical Inference, §6.3–6.5)

This repo hosts the corrected artifact generation — the binaries produced by the corrected exporters in the GitHub repo, which reproduce the paper's three headline findings. Three earlier conclusions are superseded, and the supersession is itself a reported result; the earlier binaries are retained in superseded/ as negative-result evidence.

1. State mutation, not attention, is the ANE cliff (§6.3). The stateless host-owned-cache temporal step (MRT2TemporalBodyCarry: 48 K/V caches as ordinary inputs, one-token updates as ordinary outputs, no ct.StateType) compiles the full 12-layer stack to one ANE-resident graph (costWeights=ane:1.000, p99 14.991 ms on iPhone 12 Pro). Every in-graph ct.StateType variant fails ANECCompile() with error −14. Honesty (§6.7): the shipped runtime still places temporal on .cpuAndGPU because ANE admission proved instance-fragile; the stateless graph is the proven escape to re-land.
2. Layout determines FP16 survival (§6.4). The decoder FP16 export is finite and ANE-resident after a channels-first (NCHW) rewrite plus an exact mid-network rescale. The naive channels-last FP16 export produces non-finite output (finite ratio 0.71) — "do not re-export at fp16" was wrong.
3. Weight bandwidth shapes the graph (§6.5). Twelve depth predictions per frame are weight-bandwidth-doomed no matter how little they compute, so the corrected depth body samples all 12 RVQ levels in one in-graph FP16 rollout fed by host-supplied Gumbel noise. FP32 rollout is token-exact (0/900); FP16 ships at 12.7 ms/frame on A14.

The authoritative artifact map is MODELS.md; the corrected device evidence is docs/validation-receipts.md.

Why surgery?

One graph cannot serve this model well. MRT2's generation step chains four computations with opposite hardware affinities — and the ANE compiler fails silently when you push it too far: the whole model as one graph reports success while quietly running on CPU at ~640 ms/frame, 16× over budget, no error raised. So the pipeline is cut at the joints and each piece lands on the silicon that wants it:

prompt text ──(compiled once, on a Mac)──▶ prompt vector
               │
               ▼
┌──────────────────────────────────┐
│  TEMPORAL  (carry, stateless)    │ ◀── ANE-clean (proven);
│  Predicts the next 40 ms of music│     shipped .cpuAndGPU; fp16
└──────────────┬───────────────────┘
               ▼
┌──────────────────────────────────┐
│  DEPTH  (in-graph rollout)       │ ◀── CPU/GPU/ANE (bandwidth)
│  Samples 12 RVQ levels in-graph  │     fp16, host Gumbel noise
└──────────────┬───────────────────┘
               ▼
┌──────────────────────────────────┐
│  RVQ GATHER  (Swift / C++)       │ ◀── CPU
│  Sum 12 codebook rows            │     data-dependent logic
└──────────────┬───────────────────┘
               ▼
┌──────────────────────────────────┐
│  DECODER  (SpectroStream, NCHW)  │ ◀── Neural Engine
│  Tokens → spectrogram frames     │     fp16, channels-first
└──────────────┬───────────────────┘
               ▼
┌──────────────────────────────────┐
│  iSTFT + OVERLAP-ADD  (C++)      │ ◀── CPU
│  Spectrogram → stereo PCM        │     cheap DSP
└──────────────┬───────────────────┘
               ▼
   48 kHz stereo audio — 25 frames per second

Sampling entropy (RNG, Gumbel noise) and the RVQ codebook gather stay on the host, so token-level parity is provable and seeds reproduce. CFG is baked at conditioning time — no runtime guidance machinery on-device.

What's in the download

Corrected artifacts (current)

File	Size	Precision	Compute target	Role
MRT2TemporalBodyCarry.mlpackage	349 MB	fp16	ANE-clean (proven); shipped runtime .cpuAndGPU	Stateless temporal step: 48 K/V caches in, 48 one-token updates out, host owns mutation. 2-frame bucket (--frames 2, history_length 0).
MRT2DepthBodyRollout.f16.mlpackage	71 MB	fp16	CPU/GPU/ANE (bandwidth-bound)	In-graph depth rollout: all 12 RVQ levels sampled in one prediction. The ship variant.
MRT2DepthBodyRollout.f32.mlpackage	141 MB	fp32	CPU/GPU	Token-exact reference (0/900 mismatches vs the reference sampler).
SpectroStreamDecoder.f16.mlpackage	68 MB	fp16 (fp16-safe rescale)	ANE (.cpuAndNeuralEngine)	NCHW-internal decoder — the only numerically finite FP16 variant. The ship variant.
SpectroStreamDecoder.mlpackage	136 MB	fp32	CPU/GPU	NCHW FP32 reference (SNR 118.85 dB vs MLX). Same graph as the fp16 ship variant, full precision.
SpectroStreamRVQCodebooks.f32.bin (+ .json)	12.6 MB	f32	CPU	12×1024×256 codebook table, host gather
examples/test_vector_smooth_electronic.bin (+ .json)	1 KB	f32	—	One certified conditioning vector
metadata/	—	—	—	Per-model export receipts (coremltools config, full I/O schemas)
validation/	—	—	—	Parity receipts backing every number on this card
fixtures/	1.3 MB	—	—	MLX reference tensors → validate without installing MLX

All models are mlprogram, minimum deployment target iOS 18 / macOS 15.

Device caveat, stated plainly (§6.7): the decoder's FP16 finiteness is compute-unit-dependent — on iPhone 12 Pro, .cpuAndNeuralEngine yields finite output (184,320/184,320 at 25 frames, p99 24.77 ms) while CPU-only and CPU+GPU produce non-finite output from the same artifact. Pin the fp16 decoder to .cpuAndNeuralEngine, or use the fp32 reference. And ANE admission for the temporal stack proved instance-fragile in a full app; the shipped runtime places temporal on .cpuAndGPU with the ANE-clean stateless graph as the documented escape.

Superseded artifacts (superseded/, negative-result evidence)

File	Size	Why superseded
superseded/MRT2TemporalBody.mlpackage	349 MB	Stateful (48 ct.StateType) temporal step. Every in-graph state-mutation variant fails ANECCompile −14 on device (§6.3).
superseded/MRT2DepthBody.mlpackage	93 MB	FP32 logits + 12 host-side predictions per frame — weight-bandwidth-doomed (§6.5).

Do not treat these as the paper's shipped configuration. Their export receipts and validation receipts moved with them under superseded/.

Tensor contracts (corrected artifacts)

Model	Inputs	Outputs
MRT2TemporalBodyCarry	temporal_inputs [1,2,1024] f32 · source_encoded [1,2,256] f32 · 48 × temporal_layer_XX_{self,cross}_{key,value}_cache_in [1,41,8,128] fp16	temporal_outputs [1,2,1024] · 48 × ..._cache_updates [1,2,8,128] fp16
MRT2DepthBodyRollout	temporal_frame [1,1,1024] f32 · gumbel_noise [12,1024] f32 · inverse_temperature [1] f32	sampled_codes [12] int32 · temporal_feedback [1,1024]
SpectroStreamDecoder (both precisions)	decoder_embeddings [1,25,256] f32 (host RVQ lookup sum)	decoder_stft [1,96,480,4] f32 (→ host iSTFT/overlap-add)

Everything is static-shaped. sampled_codes are codebook-local (0–1023); the unique id is 6 + level*1024 + code. temporal_feedback is the mean of the 12 sampled token embeddings (×32 scale baked in) — the next frame's temporal_inputs row. This shipped temporal package is the 2-frame bucket; re-export with --frames N for other buckets. Full I/O schemas are in metadata/.

Usage (Swift sketch)

The one-frame loop, structurally. The host owns the K/V ring buffer — the temporal graph is pure. A production runtime additionally needs a lock-free PCM ring and lookahead buffering sized from p99 inference time:

import CoreML

let temporal = try MLModel(contentsOf: temporalCarryURL, configuration: cpuGpuConfig) // shipped .cpuAndGPU; ANE-clean graph, see §6.7
let depth    = try MLModel(contentsOf: depthRolloutURL,  configuration: anyConfig)    // fp16 in-graph rollout
let decoder  = try MLModel(contentsOf: decoderF16URL,    configuration: aneConfig)    // MUST be .cpuAndNeuralEngine for fp16
var kvCaches = HostKVCaches(layers: 12, window: 41)     // 48 fp16 buffers, host-owned
let codebooks = RVQCodebooks(contentsOf: codebooksURL)  // 12×1024×256 f32 table
let source    = loadConditioning("test_vector_smooth_electronic.bin")  // [1,256]

while generating {                                      // 25 Hz
  // 1. temporal step — caches go IN as inputs, one-token updates come OUT
  let t = try temporal.prediction(from: kvCaches.asInputs(
            ["temporal_inputs": prevFrameEmbedding, "source_encoded": source]))
  kvCaches.apply(updates: t)                            // host-owned ring-buffer write
  // 2. depth rollout — host supplies Gumbel noise, graph samples all 12 levels
  let d = try depth.prediction(from: ["temporal_frame": t.lastFrame,
                                      "gumbel_noise": gumbel(seed),
                                      "inverse_temperature": invTemp])
  // 3. codebook gather on the CPU → RVQ embeddings; feedback comes from the graph
  // 4. decoder → pre-iSTFT tensor → iSTFT + overlap-add on the CPU → PCM
  prevFrameEmbedding = d.temporalFeedback
}

The Python wrappers in the GitHub repo's exporters/mrt2_coreml/ are the executable specification for each host-side stage.

Verify our numbers

Every number on this card has a receipt in validation/. Quick local check:

pip install coremltools numpy torch
git clone https://github.com/mattmireles/magenta-realtime-2-iphone && cd magenta-realtime-2-iphone
hf download mattmireles/magenta-realtime-2-iphone --local-dir hf

# Depth rollout, FP32: zero token mismatches required (gate: PASS)
PYTHONPATH=exporters python validation/validate_depth_body_rollout.py \
    --model-path hf/MRT2DepthBodyRollout.f32.mlpackage --float32

The temporal and decoder validators recompute the MLX reference from Google's checkpoint (requires the magenta_rt MLX stack on a Mac); fixtures/ contains pre-dumped MLX reference tensors for the fixture-based path. And every .mlpackage here is reproducible: run the exporters in the GitHub repo against mrt2_small.safetensors and compare — we regenerated every corrected package from the public exporters before uploading and the weight payloads matched the certified originals byte for byte.

Checksums (sha256), corrected artifacts

File	sha256
MRT2TemporalBodyCarry.mlpackage/Data/com.apple.CoreML/weights/weight.bin	02657e24a19402fed8210e431e1a67baa0c8196d4d23f0c78dd931668a9c5a64
MRT2DepthBodyRollout.f16.mlpackage/Data/com.apple.CoreML/weights/weight.bin	6c69ebae8cd7401fcb9927e3ff558d5b240cebdae2e29972d731ee08bd81c4f8
MRT2DepthBodyRollout.f32.mlpackage/Data/com.apple.CoreML/weights/weight.bin	af3ec4f9fadb84991b5dabbac58acd44294df7514226af2db63aa79659e304a9
SpectroStreamDecoder.f16.mlpackage/Data/com.apple.CoreML/weights/weight.bin	ceb3ed7d5b0286e69a937a17db253ce5a95e6bfa5560a187eb4c17de9e080f5d
SpectroStreamDecoder.mlpackage/Data/com.apple.CoreML/weights/weight.bin	38cbdf5ca97fe1744fa4d5a5caee23d50170a7eeb18daed1c40fe3a73aed9852

Conditioning

The temporal body cross-attends to a 256-dim compiled prompt (source_encoded). The exporter (export_conditioning.py) compiles any text prompt through MusicCoCa + the MRT2 conditioning encoder on a Mac, deterministically, with CFG baked at reference strength (3.0, 1.0, 1.0).

We ship exactly one conditioning vector — "smooth electronic", certified byte-reproducible — as a test vector, not a preset. We have not run the listening validation that would justify a preset library, and unvalidated presets are how on-device generation ends up sounding broken. Compile your own prompts; treat conditioning quality as experimental.

Requirements

• iOS 18+ / macOS 15+ (mlprogram floor)
• The fp16 decoder requires a device with a Neural Engine and .cpuAndNeuralEngine placement — it is non-finite on CPU and GPU. The fp32 decoder runs anywhere.
• Proven on iPhone 15 Pro Max (A17 Pro) and iPhone 12 Pro (A14 Bionic,
  2020. — both held a full 10-minute run with zero underruns and zero dropped frames. The older A14 needs a longer startup reservoir (≈15 s of PCM primed before playback) to absorb its higher per-frame tail.
• coremltools ≥ 8 to re-export

License & attribution

The model weights in this repository are content-preserving derivatives (numerical precision and memory-layout transforms only — no fine-tuning, pruning, or distillation) of google/magenta-realtime-2 by Google DeepMind, and are redistributed under the same CC-BY-4.0 license. Use of the model remains subject to the upstream model's terms of use; you are responsible for what you generate.

Conversion code, validation harnesses, and docs are Apache-2.0 in the GitHub repo.

Credits

• Google DeepMind — the Magenta team, for Magenta RealTime 2, SpectroStream, and MusicCoCa, and for releasing real on-device weights under a license that makes work like this possible. (magenta/magenta-realtime)
• Apple's coremltools team.
• Port by Matt Mireles. Prior art in the same spirit: kokoro-coreml.

Real-time music generation in your pocket, with the receipts to prove it.