mlx-community/gemma-4-12B-it-OptiQ-4bit

A 4-bit mixed-precision MLX quant produced by mlx-optiq, the sensitivity-aware quantization toolkit for Apple Silicon. Scores +6.40 over stock uniform 4-bit on the six-metric Capability Score, the second-largest mixed-precision gain in the Gemma-4 lineup.

A 4-bit mixed-precision MLX quant of google/gemma-4-12B-it, the unified (text+vision+audio) Gemma-4. This artifact is the text-inference path: the language tower is quantized and the vision/audio towers are dropped during conversion. Per-layer bit-widths come from a KL-divergence sensitivity pass on a six-domain calibration mix (prose, reasoning, code, agent, tool-call, constraint-bearing instructions). Sensitive layers go to 8-bit; robust ones stay at 4-bit.

Quantization details

Property	Value
Predominant precision	4-bit
Layers at 8-bit (sensitive)	156
Layers at 4-bit (robust)	172
Total quantized layers	328
Average bits per weight	5.22
Group size	64
Calibration mix	six-domain mix (40 samples × 6 domains)
Reference for sensitivity	uniform-4-bit (bf16 base too large to fit in RAM; auto-resolved)

We follow the same naming convention llama.cpp uses for Q4_K_M and similar mixed-precision quants: the "4-bit" label is for the predominant precision, not the weighted average. The mixed allocation is what lets this build beat stock uniform 4-bit on the Capability Score at a comparable disk size.

Usage

Load it with mlx-lm and use it as usual:

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/gemma-4-12B-it-OptiQ-4bit")
response = generate(
    model, tokenizer,
    prompt="Explain quantum computing in simple terms.",
    max_tokens=200,
)

Gemma-4-12B is a reasoning model with a thinking channel. For direct (non-thinking) answers on tasks like extraction or classification, pass chat_template_kwargs={"enable_thinking": False} when applying the chat template.

For more (mixed-precision KV-cache serving, sensitivity-aware LoRA fine-tuning, OpenAI + Anthropic-compatible inference server, hot-swap mounted adapters, sandboxed Python execution for agent workflows), install mlx-optiq:

pip install mlx-optiq

See the Gemma-4 family guide on mlx-optiq.com for sampling defaults, training recipes, and family-specific caveats.

Benchmarks

Six-metric Capability Score (mean of MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop). Apples-to-apples comparison against stock uniform 4-bit, both at the full sample counts:

Metric	OptIQ	Uniform 4-bit	Δ
MMLU (5-shot, 1000 samples)	42.6%	34.4%	+8.3
GSM8K (1000 samples, 3-shot CoT)	93.4%	90.1%	+3.3
IFEval (full set, strict)	73.9%	71.2%	+2.8
BFCL-V3 simple (200 calls)	71.0%	71.5%	−0.5
HumanEval (164 problems, pass@1)	88.4%	76.8%	+11.6
HashHop (long-context retrieval)	40.0%	27.0%	+13.0
Capability Score (mean of 6)	68.23	61.83	+6.40
KL vs uniform-4-bit reference (mean / p95)	1.52 / 4.85	—	—
On-disk size	8.3 GB	6.3 GB	+2.0

Every metric gets one equal vote. Disk size is reported next to the score as an honest second axis instead of being folded into the score. BFCL-V3 is within its ±6.3pp confidence interval (a tie); the other five benchmarks all improve, with the largest gains on long-context retrieval and code. MMLU is scored cloze-style (answer-letter log-likelihood), which underscores reasoning models that prefer to think before answering; the strong GSM8K and HumanEval results are the better quality signal. See the eval-framework writeup for the full methodology.

Links

• Project website: mlx-optiq.com
• Gemma-4 family guide: mlx-optiq.com/docs/gemma-4
• PyPI: pypi.org/project/mlx-optiq
• Calibration mix: mlx-optiq.com/blog/calibration-mix
• Eval framework: mlx-optiq.com/blog/eval-framework
• Base model: google/gemma-4-12B-it

License

Gemma license (inherits from base model). See https://ai.google.dev/gemma/terms for the terms of use.