The Mixture of Experts (MoE) architecture has stopped being just an academic technique and has started appearing in commercial, open-weight, and enterprise models. The reason is simple: it allows increasing the model's total capacity without activating all parameters for each token. In practice, this opens the way for larger models, with a better relationship between quality, inference cost, and throughput.
This e-book maps the main players who have publicly associated their models with MoE, explains how the architecture works, and translates the topic into infrastructure strategy, LLMaaS, agents, data center, and enterprise operations.
Executive reading: MoE does not automatically mean "a better model." It means a different way of scaling: part of the model becomes specialized and only a fraction is activated per token. The gain depends on training, routing, data, expert balancing, memory, network, and serving stack.
Player status: Google, Meta, Mistral, Alibaba/Qwen, DeepSeek, xAI, Databricks, Snowflake, AI21, Moonshot/Kimi, and MiniMax have public disclosures associating relevant models with MoE. OpenAI and Anthropic are treated in this material as publicly N/D (not disclosed), since they do not openly confirm the architecture of their current frontier models.
What is MoE in LLMs
In a traditional dense model, most of the relevant weights participate in processing each token. In an MoE model, certain layers are replaced by a set of experts, typically specialized feed-forward networks. A router chooses which experts should process each token, frequently using top-1, top-2, top-4, or top-k.
The most important conceptual difference is separating total parameters from active parameters. A model may have hundreds of billions or even trillions of total parameters, but activate only a fraction per token. This reduces the computational cost per inference step, although memory, network, and orchestration still need to handle the model as a distributed system.
Mental formula: dense scales by activating almost everything. MoE scales by increasing total capacity, but triggering experts depending on the token and task. Instead of "one giant brain always fully engaged," think of "a panel of experts with a router."
Core components
- Router/Gate: calculates which experts should receive each token.
- Experts: specialized sub-networks, generally MLPs/feed-forward, with independent weights.
- Top-k: number of experts selected per token.
- Load balancing: mechanism to prevent a few experts from becoming overloaded.
- Expert parallelism: technique for distributing experts across GPUs/nodes.
Timeline: from research to production
The idea of conditional computation predates modern LLMs, but it gained traction with the Sparsely-Gated Mixture-of-Experts layer proposed in 2017, which showed how to increase capacity without a proportional increase in computational cost. Later, works such as Switch Transformer simplified routing and demonstrated sparse models at trillion-parameter scale.
| Period | Milestone | Impact |
|---|---|---|
| 2017 | Sparsely-Gated MoE | Popularizes the expert layer with a trainable gate and conditional computation. |
| 2021-2022 | Switch Transformer / GShard | Simplification of routing and proof of scale in very large models. |
| 2023-2024 | Mixtral, Grok-1, DBRX, Arctic, Jamba | MoE starts appearing in open, enterprise, and commercial API models. |
| 2025-2026 | DeepSeek-V3, Qwen3, Llama 4, Kimi K2, MiniMax | The competition starts to involve active parameters, inference cost, agents, and long context. |
The critical point for companies is that MoE is not just a model architecture choice. It also changes infrastructure design: more dependencies between GPUs, greater importance of low-latency networking, the need to orchestrate experts, and increased complexity in monitoring per-token latency.
LLM x MoE Players
The table below separates models with publicly declared MoE from those whose use of MoE has not been confirmed. For closed models, the fact that there are market rumors should not be treated as technical evidence. In corporate positioning, the correct approach is to use "publicly confirmed" or "not disclosed."
| Player | Model/family | Public MoE | Disclosed parameters | Strategic reading |
|---|---|---|---|---|
| Gemini 1.5 | Yes | MoE architecture announced; parameter details not disclosed. | MoE in a large-scale, long-context commercial model. | |
| Meta | Llama 4 Scout / Maverick | Yes | Scout: 17B active, 16 experts; Maverick: 17B active, 128 experts. | Open-weight multimodal with MoE, strong for the ecosystem and integrators. |
| Mistral AI | Mixtral 8x7B / 8x22B | Yes | 8x7B: 47B total, 13B active. | Open-weight reference for efficient use and flexible deployment. |
| Alibaba / Qwen | Qwen3-235B-A22B / 30B-A3B | Yes | 235B total, 22B active; 30B total, 3B active. | Open MoE focused on reasoning, code, and multilingual capabilities. |
| DeepSeek | DeepSeek-V3 | Yes | 671B total, 37B active per token. | Example of aggressive scale with lower active cost. |
| xAI | Grok-1 | Yes | 314B parameters; 25% active per token. | Large open-weight model, relevant as a closed/open MoE reference. |
| Databricks | DBRX | Yes | 132B total, 36B active; 16 experts, chooses 4. | Enterprise MoE integrated into a data stack and corporate training. |
| Snowflake | Arctic | Yes | 480B total, 17B active; Dense-MoE hybrid. | Enterprise LLM focused on efficiency and data workloads. |
| AI21 | Jamba / Jamba 1.5 | Yes | Hybrid Transformer-Mamba-MoE architecture. | Long context and efficiency with hybrid design. |
| Moonshot AI | Kimi K2 | Yes | 1T total, 32B active. | MoE geared toward agents, code, and tool-use at open scale. |
| MiniMax | MiniMax-Text-01 / M1 | Yes | 456B total, 45.9B active. | Long context and reasoning with hybrid architecture + MoE. |
| OpenAI | GPT-4o / GPT-4.1 / others | N/D | Architecture not publicly disclosed. | Avoid claiming MoE without official confirmation. |
| Anthropic | Claude | N/D | Architecture not publicly disclosed. | Treat as a closed model with no public MoE specification. |
Note: the figures above should be used as a market snapshot based on public disclosures. For closed models, architecture rumors do not replace an official source, technical paper, model card, or repository published by the vendor.
Reading the main players
Google — Gemini 1.5
Google stated that Gemini 1.5 uses a new MoE architecture to improve training and serving efficiency. This case is strategic because it associates MoE with multimodal and long-context models in a commercial environment via Google AI Studio and Vertex AI.
Meta — Llama 4
Meta stated that Llama 4 Scout and Maverick are its first Llama models built with MoE. The combination of multimodality, open-weight, and experts places the Llama family in an important position for integrators.
Mistral — Mixtral
Mixtral 8x7B popularized open-weight MoE. The model has 47B total parameters and 13B active, becoming a didactic example of how MoE separates total capacity from active cost.
DeepSeek — V3
DeepSeek-V3 is one of the most cited examples of MoE scale: 671B total parameters and 37B activated per token. Market interest comes from the relationship between performance, cost, and an architecture open enough for technical analysis.
Qwen — Qwen3 MoE
The Qwen3 family brought dense and MoE variants. Qwen3-235B-A22B, with 235B total and 22B active, became a reference for open-weight use in reasoning, code, and multilingual applications.
Databricks and Snowflake
DBRX and Arctic show that MoE is also an enterprise topic: not just for chat, but for data, SQL, RAG, governance, custom training, and corporate models.
Moonshot/Kimi and MiniMax
Kimi K2 and MiniMax-M1/Text-01 show a strong trend in open and Asian models: MoE, long context, focus on agents, code, and tool-use with controlled active parameters.
Dense x MoE
The decision between dense and MoE models is not binary. In many environments, the best portfolio combines small dense models for low-complexity tasks, large dense models for predictability, and MoE models for high-capacity workloads, agents, code, long context, or high-throughput scenarios.
| Criterion | Dense model | MoE model |
|---|---|---|
| Cost per token | More directly proportional to the model's active size. | Can be lower than the total size would suggest, since it activates few experts. |
| Memory | Usually needs to load the complete model, but architecture and serving are simpler. | Also needs to manage many weights; memory and expert distribution matter a great deal. |
| Latency | More predictable in small batches. | Can vary due to routing, balancing, all-to-all communication, and caching. |
| Quality | Good stability and operational simplicity. | Can gain capacity and specialization, but depends on well-balanced routing and training. |
| Operation | Easier for smaller teams. | Requires a mature stack: expert parallelism, observability, and high-performance networking. |
| Best use | Classification, summarization, support, standardized tasks, edge, or lower cost. | Agents, code, reasoning, long context, high throughput, and larger models. |
Common mistake: do not sell MoE as "AI that thinks with multiple brains" without explaining cost, routing, and active parameters. The correct narrative for business is scale efficiency: more potential capacity with selective computation.
Infrastructure to run MoE
From an infrastructure standpoint, MoE is attractive because it reduces active parameters per token, but it does not eliminate the need for memory for weights, high-speed networking, and orchestration. The real cost depends on batch size, context size, weight precision, quantization, KV cache, GPU-to-GPU communication, and expert distribution.
Critical infrastructure layers
- GPU and memory: models with hundreds of billions of parameters require aggregate memory, even when few parameters are active per token.
- Low-latency network: expert parallelism can generate all-to-all communication between GPUs and nodes.
- Serving engine: vLLM, TensorRT-LLM, SGLang, TGI, and proprietary stacks need to support efficient routing.
- Quantization: FP8, INT8, INT4, and variants reduce memory, but require quality testing.
- Observability: measure tokens/s, TTFT, p95/p99 latency, expert usage, errors, cost per million tokens, and network saturation.
- Data and storage: RAG, logs, datasets, checkpoints, and objects need high-capacity storage and governance.
| Layer | Function | Point of attention |
|---|---|---|
| GPU Compute | Run experts, attention, and MLPs | VRAM, NVLink, PCIe, FP8/INT8, multi-GPU |
| Network | Synchronize experts and batches | InfiniBand/Ethernet 100/200/400G, latency, all-to-all |
| Storage | Model weights, snapshots, datasets, RAG | S3, NFS/Lustre, throughput, versioning |
| Serving | Queue, batching, cache, routing | Autoscaling, p95/p99, fallback, cost per token |
| Governance | Security, audit, LGPD, data | Per-client isolation, logs, encryption, IAM |
Implication for the data center: MoE increases the importance of high-capacity internal connectivity, predictable power, adequate cooling, a well-designed GPU cluster, and scalable storage. For LLMaaS and bare metal GPU offerings, the data center architecture becomes part of the product.
Impact for business and AI products
For the end customer, the acronym MoE only matters when it improves results, cost, latency, or capacity. The role of a technology company is to translate the architecture into concrete benefits: faster automation, more precise agents, lower inference cost, and more suitable infrastructure.
Applications with greater fit
- Corporate LLMaaS: offer models via API with cost control, isolation, and observability.
- AI agents: execute tasks with tools, memory, RAG, workflows, and governance.
- Enterprise RAG: combine MoE models with document bases, contracts, tickets, CRM, and internal data.
- Vibe coding and automation: use strong code models to accelerate development and integrations.
- Support and copilots: use routing between models: small ones for simple tasks, MoE for complex tasks.
| Offering | Commercial message | Possible deliverable |
|---|---|---|
| LLMaaS | Scalable AI with controlled cost per token. | Managed API, dashboard, limits, logs, models per profile. |
| GPU Bare Metal | Dedicated infrastructure for open and MoE models. | H100/H200/MI300 clusters, network, storage, support. |
| Enterprise RAG | Internal knowledge with security and audit. | Data pipeline, vector database, S3/NFS, LGPD policies. |
| Agents | Intelligent workflows with governance. | Agents per department, tool-use, approvals, reports. |
| AI Consulting | Technical choice between dense, MoE, and closed models. | Assessment, POC, benchmark, and production plan. |
Recommendations for EnQ Digital
EnQ Digital can use the "LLM x MoE Players" topic as authority content to connect three narratives: AI technology, data center infrastructure, and digital transformation. The differentiator lies in explaining the architecture in an executive manner while simultaneously demonstrating technical command of GPU, network, storage, private cloud, and LLMaaS.
Recommended content plan
- Post 1: "What is MoE?" — the experts and router analogy.
- Post 2: "LLM x MoE Players" — table with publicly confirmed players.
- Post 3: "Why does MoE change AI cost?" — total vs. active parameters.
- Post 4: "Infrastructure for AI" — GPU, network, storage, power, and cooling.
- Post 5: "How EnQ can help" — LLMaaS, bare metal, cloud, colocation, and connectivity.
Value proposition (key message for sales): EnQ Digital connects AI, cloud, data center, and connectivity for companies that want to turn language models into real products, with secure, scalable, and observable infrastructure.
Checklist for an MoE/LLM POC
- Define the use case: RAG, agent, support, code, document analysis, or automation.
- Select models: closed, open-weight, small dense, large dense, or MoE.
- Measure quality: accuracy, hallucination, instruction adherence, safety, and internal benchmark.
- Measure operations: p95/p99 latency, tokens/s, TTFT, cost per million tokens, and GPU usage.
- Validate governance: LGPD, isolation, logs, encryption, IAM, and data retention.
- Plan for scale: GPU capacity, storage, network, backup, observability, and support.
Essential glossary
| Term | Objective definition |
|---|---|
| MoE | Mixture of Experts: architecture that divides parts of the model into experts and activates only some per token. |
| Router/Gate | Component that decides which experts process each token. |
| Total parameter | Total number of model weights, including experts that may not be activated for each token. |
| Active parameter | Number of parameters used to process a token or input in a pass. |
| Top-k | Number of experts chosen by the router per token. |
| Expert parallelism | Distribution of experts across GPUs or servers to enable training/inference. |
| All-to-all | Communication pattern frequent in MoE when tokens need to be sent to different experts. |
| KV cache | Memory used to accelerate generation in autoregressive models; grows with context and batch. |
| Dense model | Model in which most of the relevant network is uniformly triggered at each token. |
Sources and recommended reading
- Shazeer et al. (2017) — Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538.
- Fedus, Zoph & Shazeer (2021/2022) — Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961 / JMLR.
- Google — Introducing Gemini 1.5, Google's next-generation AI model. Official Google blog.
- Mistral AI — Mixtral 8x7B model card and "Mixtral of Experts." Docs and official Mistral publication.
- DeepSeek — Introducing DeepSeek-V3 and DeepSeek-V3 Technical Report.
- Qwen / Alibaba — Qwen3 blog, Qwen3 Technical Report, and Qwen3-235B-A22B model cards.
- Meta — The Llama 4 herd: natively multimodal AI innovation. Official Meta AI blog and Llama 4 model cards.
- xAI — Open Release of Grok-1 and xai-org/grok-1 repository.
- Databricks — Introducing DBRX: A New State-of-the-Art Open LLM. Official Databricks blog.
- Snowflake — Snowflake Arctic: enterprise-grade LLM and dense-MoE hybrid architecture.
- AI21 — Jamba: Hybrid Transformer-Mamba MoE architecture. AI21 research/blog.
- Moonshot AI — Kimi K2 technical report, GitHub/Hugging Face, and Kimi API platform.
- MiniMax — MiniMax-Text-01 / MiniMax-M1 repositories, model cards, and technical reports.
- OpenAI — GPT-4 Technical Report: architecture and training details not disclosed in the public report.
Verification note: the player map should be reviewed periodically. Vendors change nomenclature, availability, pricing, context, endpoints, and licenses. For a commercial proposal, always validate the model card and the vendor's current terms of use.