arXiv cs.LG论文
文档敏感度分类中经典与Transformer模型的基准测试
本文针对组织文档自动敏感度分类中普遍存在的标签泄漏问题，构建了受控语料库Strategic 16K（包含16,000份维基解密外交电文），并系统评估了六种模型。在去除三类残留标记后，BERT表现最佳（准确率89.14%，F1 89.33%），经典模型TF-IDF+逻辑回归在较低计算成本下也具竞争力。该研究首次提供了在明确泄漏控制条件下可复现的敏感度分类基准。
这篇是正式发表的长论文，站内提供中文解读，全文请到原文阅读 PDF。
Abstract:Automatic sensitivity classification of organizational documents is a critical yet underserved problem, where the consequences of misclassification range from regulatory violations to security breaches. While AI-based approaches offer a scalable alternative to manual review, their reliability depends fundamentally on the integrity of training data. A pervasive but underreported problem in this domain is label leakage: residual classification markers embedded within document bodies that allow models to exploit surface shortcuts rather than learning genuine content-based sensitivity signals, producing performance estimates that are inflated and unreliable. This paper addresses this problem by introducing Strategic 16K, a carefully constructed, leakage-controlled corpus of 16,000 diplomatic cables sourced from the WikiLeaks Public Library of US Diplomacy (PlusD), and presents a systematic benchmark evaluating six model architectures spanning classical machine learning and transformer-based approaches. We document an extended leakage removal protocol that identifies and eliminates three categories of residual classification markers embedded within document bodies. On the clean benchmark, BERT achieves the strongest performance (Accuracy = 89.14%, F1 = 89.33%), followed by ELECTRA (Accuracy = 88.57%, F1 = 88.90%). Among classical models, TF-IDF with Logistic Regression achieves the strongest performance at significantly lower computational cost. These results constitute the first fully reproducible sensitivity classification benchmark constructed under explicit leakage-controlled conditions from WikiLeaks PlusD.