A Bayesian Approach to Measuring Evidence in L2 Research

An Empirical Investigation

Reza Norouzian, Michael De Miranda, Luke D Plonsky

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Null hypothesis testing has long since been the ‘go-to analytic approach’ in quantitative second language (L2) research (Norris, 2015, p. 97). To many, however, years of reliance on this approach has resulted in a crisis of inference across the social and behavioral sciences (e.g., Rouder et al., 2016). As an alternative to the null hypothesis testing approach, many such experts recommend the Bayesian hypothesis testing approach. Adopting an open-science framework, the present study (a) re-evaluates the empirical findings of 418 t-tests from published L2 research using Bayesian hypothesis testing, and (b) compares the Bayesian results with their conventional, null hypothesis testing counterparts as observed in the original reports. The results show that the Bayesian and the null hypothesis testing approaches generally arrive at similar inferential conclusions. However, considerable differences arise in the rejections of the null hypothesis. Notably, in 64.06% of cases when p-values fell between.01 and.05 (i.e., evidence to reject the null), the Bayesian analysis found the evidence in the primary studies to be only at an ‘anecdotal’ level (i.e., insufficient evidence to reject the null). Practical implications, field-wide recommendations, and an introduction to free online software (https://rnorouzian.shinyapps.io/bayesian-t-tests) for Bayesian hypothesis testing are discussed.

Original languageEnglish (US)
Pages (from-to)248-261
Number of pages14
JournalModern Language Journal
Volume103
Issue number1
DOIs
StatePublished - Mar 1 2019

Fingerprint

hypothesis testing
evidence
behavioral science
Hypothesis Testing
social science
Null Hypothesis
expert
language
science
Values

Keywords

  • Bayes factor
  • Bayesian hypothesis testing
  • null hypothesis testing
  • open science framework
  • p-value
  • quantitative L2 research

ASJC Scopus subject areas

  • Language and Linguistics
  • Linguistics and Language

Cite this

A Bayesian Approach to Measuring Evidence in L2 Research : An Empirical Investigation. / Norouzian, Reza; Miranda, Michael De; Plonsky, Luke D.

In: Modern Language Journal, Vol. 103, No. 1, 01.03.2019, p. 248-261.

Research output: Contribution to journalArticle

Norouzian, Reza ; Miranda, Michael De ; Plonsky, Luke D. / A Bayesian Approach to Measuring Evidence in L2 Research : An Empirical Investigation. In: Modern Language Journal. 2019 ; Vol. 103, No. 1. pp. 248-261.
@article{eb4f61dc5fa043d7b1b502c2310828ff,
title = "A Bayesian Approach to Measuring Evidence in L2 Research: An Empirical Investigation",
abstract = "Null hypothesis testing has long since been the ‘go-to analytic approach’ in quantitative second language (L2) research (Norris, 2015, p. 97). To many, however, years of reliance on this approach has resulted in a crisis of inference across the social and behavioral sciences (e.g., Rouder et al., 2016). As an alternative to the null hypothesis testing approach, many such experts recommend the Bayesian hypothesis testing approach. Adopting an open-science framework, the present study (a) re-evaluates the empirical findings of 418 t-tests from published L2 research using Bayesian hypothesis testing, and (b) compares the Bayesian results with their conventional, null hypothesis testing counterparts as observed in the original reports. The results show that the Bayesian and the null hypothesis testing approaches generally arrive at similar inferential conclusions. However, considerable differences arise in the rejections of the null hypothesis. Notably, in 64.06{\%} of cases when p-values fell between.01 and.05 (i.e., evidence to reject the null), the Bayesian analysis found the evidence in the primary studies to be only at an ‘anecdotal’ level (i.e., insufficient evidence to reject the null). Practical implications, field-wide recommendations, and an introduction to free online software (https://rnorouzian.shinyapps.io/bayesian-t-tests) for Bayesian hypothesis testing are discussed.",
keywords = "Bayes factor, Bayesian hypothesis testing, null hypothesis testing, open science framework, p-value, quantitative L2 research",
author = "Reza Norouzian and Miranda, {Michael De} and Plonsky, {Luke D}",
year = "2019",
month = "3",
day = "1",
doi = "10.1111/modl.12543",
language = "English (US)",
volume = "103",
pages = "248--261",
journal = "Modern Language Journal",
issn = "0026-7902",
publisher = "Wiley-Blackwell",
number = "1",

}

TY - JOUR

T1 - A Bayesian Approach to Measuring Evidence in L2 Research

T2 - An Empirical Investigation

AU - Norouzian, Reza

AU - Miranda, Michael De

AU - Plonsky, Luke D

PY - 2019/3/1

Y1 - 2019/3/1

N2 - Null hypothesis testing has long since been the ‘go-to analytic approach’ in quantitative second language (L2) research (Norris, 2015, p. 97). To many, however, years of reliance on this approach has resulted in a crisis of inference across the social and behavioral sciences (e.g., Rouder et al., 2016). As an alternative to the null hypothesis testing approach, many such experts recommend the Bayesian hypothesis testing approach. Adopting an open-science framework, the present study (a) re-evaluates the empirical findings of 418 t-tests from published L2 research using Bayesian hypothesis testing, and (b) compares the Bayesian results with their conventional, null hypothesis testing counterparts as observed in the original reports. The results show that the Bayesian and the null hypothesis testing approaches generally arrive at similar inferential conclusions. However, considerable differences arise in the rejections of the null hypothesis. Notably, in 64.06% of cases when p-values fell between.01 and.05 (i.e., evidence to reject the null), the Bayesian analysis found the evidence in the primary studies to be only at an ‘anecdotal’ level (i.e., insufficient evidence to reject the null). Practical implications, field-wide recommendations, and an introduction to free online software (https://rnorouzian.shinyapps.io/bayesian-t-tests) for Bayesian hypothesis testing are discussed.

AB - Null hypothesis testing has long since been the ‘go-to analytic approach’ in quantitative second language (L2) research (Norris, 2015, p. 97). To many, however, years of reliance on this approach has resulted in a crisis of inference across the social and behavioral sciences (e.g., Rouder et al., 2016). As an alternative to the null hypothesis testing approach, many such experts recommend the Bayesian hypothesis testing approach. Adopting an open-science framework, the present study (a) re-evaluates the empirical findings of 418 t-tests from published L2 research using Bayesian hypothesis testing, and (b) compares the Bayesian results with their conventional, null hypothesis testing counterparts as observed in the original reports. The results show that the Bayesian and the null hypothesis testing approaches generally arrive at similar inferential conclusions. However, considerable differences arise in the rejections of the null hypothesis. Notably, in 64.06% of cases when p-values fell between.01 and.05 (i.e., evidence to reject the null), the Bayesian analysis found the evidence in the primary studies to be only at an ‘anecdotal’ level (i.e., insufficient evidence to reject the null). Practical implications, field-wide recommendations, and an introduction to free online software (https://rnorouzian.shinyapps.io/bayesian-t-tests) for Bayesian hypothesis testing are discussed.

KW - Bayes factor

KW - Bayesian hypothesis testing

KW - null hypothesis testing

KW - open science framework

KW - p-value

KW - quantitative L2 research

UR - http://www.scopus.com/inward/record.url?scp=85061001909&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85061001909&partnerID=8YFLogxK

U2 - 10.1111/modl.12543

DO - 10.1111/modl.12543

M3 - Article

VL - 103

SP - 248

EP - 261

JO - Modern Language Journal

JF - Modern Language Journal

SN - 0026-7902

IS - 1

ER -