arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Social Sciences · Sociology and Political Science

A type-token identity in the Simon-Yule model of text

Ye-Sho Chen, Ferdinand F. Leimkuhler

· Journal of the American Society for Information Science

See on the map

Abstract

There are three significant results in this paper. First, we establish a type-token identity relating the type-token ratio and the bilogarithmic type-token ratio. The plays of Shakespeare and other interesting texts serve as demonstrative examples. Second, the Simon-Yule model of Zipf's law is used to derive the type-token identity and provide a promising statistical model of text generation. Third, a realistic refinement of the Simon-Yule model is made to allow for a decreasing entry rate of new words. Simulation methods are used to show that the type-token identity is preserved with this change in assumptions. © 1989 John Wiley & Sons, Inc.

More like this