arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Computer Science · Artificial Intelligence

Chinese and Japanese Word Segmentation Using Word-level and Character-level Information

Tetsuji Nakagawa, Yūji Matsumoto

· IPSJ journal

See on the map

Abstract

In this paper, we present a hybrid method for Chinese and Japanese word segmentation. Word-level information is useful for analysis of known words, while character-level information is useful for analysis of unknown words, and the method utilizes both these two types of information in order to effectively handle known and unknown words. Experimental results show that this method achieves high overall accuracy in Chinese and Japanese word segmentation. 1

More like this