Lexicon Discovery for Language Preservation using Unsupervised Word Segmentation with Pitman-Yor Language Models (FGNT-2015-01)

Abstract

In this paper we show that recently developed algorithms for unsupervised word segmentation can be a valuable tool for the documentation of endangered languages. We applied an unsupervised word segmentation algorithm based on a nested Pitman-Yor language model to two austronesian languages, Wooi and Waima’a. The algorithm was then modified and pa-rameterized to cater the needs of linguists for high precision of lexical discovery: We obtained a lexicon precision of of 69.2% and 67.5 % for Wooi and Waima’a, respectively, if single-letter words and words found less than three times were discarded. A comparison with an English word segmentation task showed comparable performance, verifying that the assumptions under-lying the Pitman-Yor language model, the universality of Zipf’s law and the power of n-gram structures, do also hold for lan-guages as exotic as Wooi and Waima’a. Index Terms: unsupervised vocabulary acquisition, unsuper-vised language acquisition, unsupervised word segmentation