arXiv cs.CL论文
分数衰减KV缓存：面向对话系统的所有权感知内存管理以提升推理相关性
本文提出分数衰减KV缓存（FD-KVC），一种用于Transformer对话系统的新颖KV缓存管理算法。现有方法对所有缓存条目一视同仁或采用粗粒度驱逐策略，难以适应对话主题变化。FD-KVC为每个KV对维护双通道评分：累积注意力通道跟踪总体重要性，以及由时间衰减和强化学习启发的更新驱动的近期加权相关性通道。结合两者，FD-KVC既能保留历史重要标记，又能快速适应主题转移。通过所有权损失函数驱动的自适应学习率确保收敛。FD-KVC完全在CPU上运行，开销可忽略。在五个多轮对话场景（每个600个对话）中，FD-KVC在复合后期对齐上比最先进的H2O基线提升6.7%，在主题转移、渐进演化和混合主题对话上分别提升127%、87%和30%。FD-KVC适应新主题的速度比H2O快3.6倍，并达到最高的主题多样性（80.6%）。消融研究证实了各组成部分的贡献。
这篇是正式发表的长论文，站内提供中文解读，全文请到原文阅读 PDF。
Abstract:Key-value (KV) caching is essential for efficient autoregressive inference in transformer based dialog systems, yet existing strategies treat all cached entries uniformly or apply coarse eviction heuristics that fail to adapt as dialog topics evolve. We propose Fractional Decay KV-Cache (FD-KVC), a novel algorithm that maintains a dual-channel scoring mechanism for each cached KV pair: a cumulative attention channel that tracks aggregate importance (akin to H2O), and a recency-weighted relevance channel governed by temporal decay and reinforcement-inspired updates. The combination enables FD-KVC to both preserve historically important tokens and rapidly adapt when dialog topics shift. An adaptive learning rate driven by an ownership loss function ensures convergence without oscillation. FD-KVC operates entirely on CPU with negligible overhead. Across five diverse multi-turn dialog scenarios with 600 dialogs each, FD-KVC outperforms H2O, the state-of-the-art heavy-hitter baseline, by +6.7% on composite late-turn alignment, with improvements of +127% on topic-shift, +87% on gradual evolution, and +30% on mixed-topic dialogs. FD-KVC adapts to new topics 3.6X faster than H2O and achieves the highest topic diversity (80.6%) across all methods. Ablation studies confirm the contribution of each component.