Abstract:In this paper, the question of Chinese automatic segmentation is discussed using overlaying ambiguity examining method and statistics language model. The multi-time iterative method is applied to train language model, which can produce a better model. The process of training language model is described in detail. The result shows that the perplexity of language model is reduced. The accuracy of segmentation changes with different language model and the reason is analyzed.
王显芳;杜利民. 利用覆盖歧义检测法和统计语言模型进行汉语自动分词[J]. 电子与信息学报, 2003, 25(9): 1168-1173 .
Wang Xianfang; Du Limin. Automatic Segmentation of Chinese using overlaying ambiguity examining method and statistics language model. , 2003, 25(9): 1168-1173 .