Distributed AI & GPU/ACM Digital Library/2023 / 2025 Revised
Efficient Memory Management for Large Language Model Serving with PagedAttention
Woosuk Kwon, Zhuohan Li, Siyuan Shen et al.Sky Computing Lab, UC Berkeley (SOSP 2023)
High-throughput LLM serving systems are bottlenecked by KV cache memory allocation. PagedAttention adapts virtual memory paging principles to dynamically partition attention key-value states into non-contiguous physical blocks, eliminating external fragmentation and enabling memory sharing across parallel sampling requests.
Core Architectural Insights
Virtual Memory Paging
Partitions dynamic KV cache states into non-contiguous fixed blocks matching OS virtual memory pages.
Near-Zero Fragmentation
Shrinks memory waste from traditional 60% to 80% reservations down to under 4% across batch workloads.
Parallel Generation Sharing
Enables zero-copy state reuse for parallel sampling, speculative decoding and shared prompt prefixes.
Xephix Production IntegrationPagedAttention memory virtualization primitives are running in our production high-throughput GPU inference clusters.