MiniMax M3 Sparse Attention Delivers 15.6× Speed Boost for Long‑Context AI

MiniMax M3 sparse attention promises 15.6× decoding speed gain

MiniMax has released a detailed technical report that not only dissects the successes of its M2 series but also teases the upcoming MiniMax M3 sparse attention model. Leveraging a custom sub‑quadratic attention framework, the design reportedly achieves ↑ 15.6x faster decoding on one‑million‑token contexts, a leap that could make ultra‑long‑context AI agents economically practical.

Why full‑quadratic attention stalls at scale

Traditional full‑attention scales quadratically, forcing each token to interact with every other token—a cost that explodes with longer inputs. Past experiments with sliding‑window or linear attention compromised multi‑hop reasoning, prompting MiniMax to retain full attention for M2 despite its hardware appetite.

“Beyond benchmarks, MiniMax’s work on MoE efficiency and agent‑oriented design is impressive,” noted Adina Yakup of Hugging Face.

The new MSA (MiniMax Sparse Attention) operates on a standard Grouped Query Attention backbone but selects blocks of real key‑value pairs rather than compressed representations, sidestepping the precision loss seen in competing methods. Early profiling suggests a ↑ 9.7x reduction in prefilling latency and the headline 15.6× decoding acceleration.

For enterprises eyeing in‑house model fine‑tuning, the M2 report supplies a blueprint for MoE routing, sigmoid gating, and expert‑specific bias terms, all released under permissive open‑source licenses. The insight aligns with broader industry moves, as noted by Reuters, to democratize high‑performance LLMs.