AMD unveiled Instella‑MoE‑16B‑A3B, a 16B parameter open‑source Mixture‑of‑Experts language model that activates only 2.8B parameters per token, trained from scratch on Instinct MI300X and MI325X GPUs.

AMD has announced the release of Instella‑MoE‑16B‑A3B, a 16‑billion‑parameter open‑source Mixture‑of‑Experts (MoE) language model that dynamically activates only 2.8 billion parameters per token, dramatically reducing inference cost while preserving high‑quality output.

Key Architecture Features

Instella‑MoE‑16B‑A3B follows a sparse‑activation design where a gating network selects a subset of expert layers for each token. This approach allows the model to scale to 16 B total parameters while keeping the active compute footprint comparable to a 3 B dense model.

The model is built on AMD’s custom transformer stack, leveraging the Instinct MI300X and MI325X GPUs for training. The training pipeline exploits the GPUs’ high‑bandwidth memory and matrix‑core optimizations to accelerate the expert routing and weight updates.

Training Details

Training was conducted from scratch on a cluster of 64 MI300X and 32 MI325X units, using a mixture of publicly available text corpora and domain‑specific datasets. The process employed a token‑level expert capacity of eight, meaning each token is processed by eight expert feed‑forward networks before the gating decision finalizes the active subset.

AMD reports that the model converged after roughly 400 billion tokens, achieving perplexity scores on par with comparable dense models of similar size, while offering up to a 70 % reduction in FLOPs during inference.

Open‑Source Availability

The full model weights, training scripts, and inference utilities are released under the Apache 2.0 license on AMD’s public GitHub repository. Developers can fine‑tune the model on downstream tasks using the provided LoRA adapters, and the repository includes Docker images pre‑configured for both AMD and NVIDIA hardware.

  • Model size: 16 B total parameters
  • Active parameters per token: 2.8 B
  • Supported GPUs: Instinct MI300X, MI325X, and compatible ROCm drivers
  • License: Apache 2.0

By making Instella‑MoE‑16B‑A3B openly accessible, AMD aims to foster a broader ecosystem of AI research and deployment that is not tied to proprietary cloud services, encouraging community contributions and transparent benchmarking.

MarkTechPost coverage of AMD Instella‑MoE‑16B‑A3B