Zero-inflated beta distribution applied to word frequency and lexical dispersion in corpus linguistics

Brent D Burch, Jesse Egbert

Research output: Contribution to journalArticle

Abstract

Corpus linguistics is the study of language as expressed in a body of texts or documents. The relative frequency of a word within a text and the dispersion of the word across the collection of texts provide information about the word's prominence and diffusion, respectively. In practice, people tend to use a relatively small number of words in a language's inventory of words and thus a large number of words in the lexicon are rarely employed. The zero-inflated beta distribution enables one to model the relative frequency of a word in a text since some texts may not even contain the word under study. In this paper, the expectation of a word's prominence and dispersion are defined under the zero-inflated beta model. Estimates of a word's prominence and dispersion are computed for words in the British National Corpus 1994 (BNC), a 100 million word collection of written and spoken language of a wide range of British English. The relationship between a word's prominence and dispersion is discussed as well as measures that are functions of both prominence and dispersion.

Original languageEnglish (US)
JournalJournal of Applied Statistics
DOIs
StatePublished - Jan 1 2019

Fingerprint

Zero Distribution
Beta distribution
Text
Corpus
Linguistics
Tend
Zero
Model
Estimate
Range of data
Language

Keywords

  • British National Corpus
  • mixture distribution
  • ranking words
  • word usage
  • zero-inflated beta distribution

ASJC Scopus subject areas

  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Cite this

@article{f4d849ca4f364e60a25d106b8d18e663,
title = "Zero-inflated beta distribution applied to word frequency and lexical dispersion in corpus linguistics",
abstract = "Corpus linguistics is the study of language as expressed in a body of texts or documents. The relative frequency of a word within a text and the dispersion of the word across the collection of texts provide information about the word's prominence and diffusion, respectively. In practice, people tend to use a relatively small number of words in a language's inventory of words and thus a large number of words in the lexicon are rarely employed. The zero-inflated beta distribution enables one to model the relative frequency of a word in a text since some texts may not even contain the word under study. In this paper, the expectation of a word's prominence and dispersion are defined under the zero-inflated beta model. Estimates of a word's prominence and dispersion are computed for words in the British National Corpus 1994 (BNC), a 100 million word collection of written and spoken language of a wide range of British English. The relationship between a word's prominence and dispersion is discussed as well as measures that are functions of both prominence and dispersion.",
keywords = "British National Corpus, mixture distribution, ranking words, word usage, zero-inflated beta distribution",
author = "Burch, {Brent D} and Jesse Egbert",
year = "2019",
month = "1",
day = "1",
doi = "10.1080/02664763.2019.1636941",
language = "English (US)",
journal = "Journal of Applied Statistics",
issn = "0266-4763",
publisher = "Routledge",

}

TY - JOUR

T1 - Zero-inflated beta distribution applied to word frequency and lexical dispersion in corpus linguistics

AU - Burch, Brent D

AU - Egbert, Jesse

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Corpus linguistics is the study of language as expressed in a body of texts or documents. The relative frequency of a word within a text and the dispersion of the word across the collection of texts provide information about the word's prominence and diffusion, respectively. In practice, people tend to use a relatively small number of words in a language's inventory of words and thus a large number of words in the lexicon are rarely employed. The zero-inflated beta distribution enables one to model the relative frequency of a word in a text since some texts may not even contain the word under study. In this paper, the expectation of a word's prominence and dispersion are defined under the zero-inflated beta model. Estimates of a word's prominence and dispersion are computed for words in the British National Corpus 1994 (BNC), a 100 million word collection of written and spoken language of a wide range of British English. The relationship between a word's prominence and dispersion is discussed as well as measures that are functions of both prominence and dispersion.

AB - Corpus linguistics is the study of language as expressed in a body of texts or documents. The relative frequency of a word within a text and the dispersion of the word across the collection of texts provide information about the word's prominence and diffusion, respectively. In practice, people tend to use a relatively small number of words in a language's inventory of words and thus a large number of words in the lexicon are rarely employed. The zero-inflated beta distribution enables one to model the relative frequency of a word in a text since some texts may not even contain the word under study. In this paper, the expectation of a word's prominence and dispersion are defined under the zero-inflated beta model. Estimates of a word's prominence and dispersion are computed for words in the British National Corpus 1994 (BNC), a 100 million word collection of written and spoken language of a wide range of British English. The relationship between a word's prominence and dispersion is discussed as well as measures that are functions of both prominence and dispersion.

KW - British National Corpus

KW - mixture distribution

KW - ranking words

KW - word usage

KW - zero-inflated beta distribution

UR - http://www.scopus.com/inward/record.url?scp=85068589117&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85068589117&partnerID=8YFLogxK

U2 - 10.1080/02664763.2019.1636941

DO - 10.1080/02664763.2019.1636941

M3 - Article

JO - Journal of Applied Statistics

JF - Journal of Applied Statistics

SN - 0266-4763

ER -