Frontier of LLM Inference: The Engineering Blueprint for KV Cache Optimization, Low-Precision Kernels, and Cluster Sizing

Description: The Definitive Systems Engineering Blueprint for Frontier LLM Inference

By 2026, computing has inverted: over 92% of enterprise GPU cycles and 85% of infrastructure budgets are consumed during inference as models transition to test-time reasoning, dynamic tree search, and trillion-token agentic workflows.

Written by Ken Huang (Adjunct Professor, University of San Francisco; CEO & CAIO, DistributedApps.ai) with technical review by Manish Bhatt (Application Security Tech Lead for AI & Crypto, Amazon Leo), The Frontier of LLM Inference provides a first-principles, mathematically rigorous master guide for building and operating industrial LLM inference engines.

• Chapter 1 (Physics of LLM Inference): Roofline models, memory walls, arithmetic intensity collapse (1-2 FLOP/byte), H100 vs. B200 limits, and continuous batching.
• Chapter 2 (KV Cache Frontier): Multi-Head Latent Attention (MLA), 93.3% latent memory compression, SGLang RadixAttention, and cross-layer KVShare.
• Chapter 3 (Speculative Decoding): Multi-Token Prediction (MTP), EAGLE-2 dynamic trees, IndexShare, and Leviathan-Chen proofs.
• Chapter 4 (Extreme Quantization): Blackwell FP4 microscaling, Native FP8 GEMM, Unsloth Triton primitives, and KIVI 2-bit asymmetric KV quantization.
• Chapter 5 (Attention Kernels): FlashAttention-3 Warp Specialization, FlashDecoding split-KV, SageAttention integer speedups, and TPU Pallas kernels.
• Chapter 6 (Disaggregated Serving): Mooncake 4-tier RDMA transfer engines, DistServe prefill-decode queue decoupling, and vLLM V1 C++ core engine.
• Chapter 7 (Serving Mega-MoE): Colossus interconnects, DeepSeek-V3 671B MoE serving, Expert Parallelism (EP), and DualPipe overlap.
• Chapter 8 (Test-Time Compute): DeepSeek-V4-Pro, GLM-5.3, Kimi K3, dynamic thinking budgets, Monte Carlo Tree Search (MCTS), and PRM verifiers.
• Chapter 9 (Ultra-Long Context): Alibaba Dual-Chunk Attention (DCA), YaRN spectral RoPE interpolation, and dynamic sparse routing for 1M+ context.
• Chapter 10 (Constrained Decoding & Production): FSM Grammars, XGrammar bitmasks, Edge SLMs, industrial SLO engineering (TTFT, TPOT, ITL), and cluster sizing.
• Chapter 11 (Token Efficiency & Software-Factory Economics): Provider prompt-cache multipliers, gateway affinity routing, prefix compression, and software-factory unit economics.
• Chapter 12 (Agentic Inference Systems): Streaming tool-call dispatch, session-native KV retention, subagent isolation, idle prefill/decode decoupling, and multi-agent session SLOs.

• Volume: 481 Pages of production engineering synthesis.
• Visuals & Artifacts: 187 architectural diagrams and system flowcharts.
• Master Index: Comprehensive Technical & Subject Index (A-Z).
• Comprehension & Solutions: 240 technical review problems with detailed engineering solutions (20 per chapter across all 12 chapters).

Target Audience: Systems Architects, Principal Machine Learning Engineers, GPU Kernel Developers, Chief AI Officers (CAIOs), Infrastructure Directors, and Graduate Researchers in High-Performance Computing.