arXiv cs.CL论文
NE-BERT：面向印度东北九种语言的多语言模型
NE-BERT是一个针对印度东北部九种低资源语言的多语言编码器模型，在约830万句子上训练，采用加权数据采样和自定义SentencePiece Unigram分词器。相比IndicBERT-V2和MuRIL，平均困惑度分别降低15.97倍和7.64倍，分词效率比mBERT提升1.5倍。在下游词性标注任务上验证了实用性，并开源了模型、测试集和训练语料（CC-BY-4.0）。
这篇是正式发表的长论文，站内提供中文解读，全文请到原文阅读 PDF。
Abstract:Large pretrained language models have demonstrated remarkable capabilities across diverse languages, yet critically underrepresented low-resource languages remain marginalized. We present NE-BERT, a domain-specific multilingual encoder model trained on approximately 8.3 million sentences spanning 9 Northeast Indian languages and 2 anchor languages (Hindi, English), a linguistically diverse region with minimal representation in existing multilingual models. By employing weighted data sampling and a custom SentencePiece Unigram tokenizer, NE-BERT outperforms IndicBERT-V2 and MuRIL across all 9 Northeast Indian languages, achieving 15.97X and 7.64X lower average perplexity respectively, with 1.50X better tokenization fertility than mBERT. We address critical vocabulary fragmentation issues in extremely low-resource languages such as Pnar (1,002 sentences) and Kokborok (2,463 sentences) through aggressive upsampling strategies. Downstream evaluation on part-of-speech tagging validates practical utility on three Northeast Indian languages. We release NE-BERT, test sets, and training corpus under CC-BY-4.0 to support NLP research and digital inclusion for Northeast Indian communities.