SemiAnalysis· @SemiAnalysis_ · X·· 6 天前AI 评分47
AI 导读
ALERT🚨: 通过在 12-16 个并发智能体会话下启用 DRAM KVCache 卸载优化，可缓存的 token 数量可增加 20-40%🚀。一块 B200 只有 180 GB HBM，扣除权重存储后，每颗芯片只剩 100 GB HBM 用于 KVCache。 这对长上下文智能体推理工作负载来说是不够的。因此，借助 SGLang HiCache，你还可以将 KVCache 卸载到 CPU 的 DDR5/LPDDR5X 内存，这有助于提升性能。(1/2)🧵
AI 生成摘要 · 以原文为准
正文
ALERT🚨: The number of tokens that can be cached can increase by 20-40%🚀 by enabling DRAM KVCache offloading optimization at 12-16 concurrent agentic sessions. A B200 only has 180 GB of HBM, and after weight storage, only 100 GB of HBM per chip remains for KVCache.
THIS IS NOT ENOUGH FOR LONG-CONTEXT AGENTIC INFERENCE WORKLOADS. Thus, with SGLang HiCache, you can also offload KVCache to the CPU's DDR5/LPDDR5X memory, which helps improve performance. (1/2)🧵
来源：SemiAnalysis · x.com