arXiv cs.CL论文
SuTRA：具有词根感知的结构统一分词
现有子词分词器优化统计压缩但忽略形态结构，尤其对词根与词缀的关系处理不当，这在形态丰富的印度语言中尤为有害，因为其基本单位是复杂的正字法音节（aksharas）而非字母。基于频率的方法过度切分单词，任意拆分词根和词缀，作者称之为“形态破碎”。为此，他们提出SuTRA，一种形态感知算法，保持akshara的不可分割性，并惩罚跨越形态边界的合并。他们还发布了印地语、马拉地语和古吉拉特语的形态分割数据集。SuTRA减少了破碎现象，在形态对齐（边界F1）上最高提升14.7%，在印地语的语义可恢复性上提升34%，并在机器翻译中平均提升8.08 chrF2。
这篇是正式发表的长论文，站内提供中文解读，全文请到原文阅读 PDF。
Abstract:Existing subword tokenizers optimize statistical compression but ignore morphological structure, particularly the relationship between roots and affixes. This is harmful for morphologically rich Indic languages, where basic units are complex orthographic syllables (aksharas) rather than letters. Frequency-based methods over-fragment words, arbitrarily splitting roots and affixes - a phenomenon we term Morphological Shattering. We propose SuTRA (Structurally-Unified Tokenization with Root Awareness), a morphology-aware algorithm that preserves akshara indivisibility and penalizes merges crossing morphological boundaries. We also release a new morphological segmentation dataset for Hindi, Marathi, and Gujarati. SuTRA reduces shattering, achieving peak gains of +14.7% in morphological alignment (Boundary F1) and +34% in semantic recoverability (Hindi) over BPE. These structural gains yield an average improvement of +8.08 chrF2 in machine translation.