Zero-inflated beta distribution applied to word frequency and lexical dispersion in corpus linguistics

Brent D Burch, Jesse Egbert

Research output: Contribution to journalArticle

Abstract

Corpus linguistics is the study of language as expressed in a body of texts or documents. The relative frequency of a word within a text and the dispersion of the word across the collection of texts provide information about the word's prominence and diffusion, respectively. In practice, people tend to use a relatively small number of words in a language's inventory of words and thus a large number of words in the lexicon are rarely employed. The zero-inflated beta distribution enables one to model the relative frequency of a word in a text since some texts may not even contain the word under study. In this paper, the expectation of a word's prominence and dispersion are defined under the zero-inflated beta model. Estimates of a word's prominence and dispersion are computed for words in the British National Corpus 1994 (BNC), a 100 million word collection of written and spoken language of a wide range of British English. The relationship between a word's prominence and dispersion is discussed as well as measures that are functions of both prominence and dispersion.

Original languageEnglish (US)
JournalJournal of Applied Statistics
DOIs
StatePublished - Jan 1 2019

    Fingerprint

Keywords

  • British National Corpus
  • mixture distribution
  • ranking words
  • word usage
  • zero-inflated beta distribution

ASJC Scopus subject areas

  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Cite this