Statistical assumptions in L2 research: A systematic review

Yuhang Hu, Luke Plonsky

Research output: Contribution to journalReview article

Abstract

Statistical tests carry with them a number of assumptions that must be checked. Failing to do so and to report the results of such preliminary analyses introduce a potential threat to the internal validity of a study and to our ability as consumers to put faith in study findings. This article systematically examines the reporting of checks on assumptions in two major journals of second-language (L2) research (Language Learning and Second Language Research) over a span of six years (2012–2017 inclusive). Using synthetic techniques, each study in the sample (K = 107) was coded for whether or not the assumptions associated with five statistical tests were reported: independent samples t-test, one-way ANOVA, Pearson correlation, multiple regression, and chi-square. In doing so, two coding standards (‘stringent’ and ‘lenient’) were adopted to arrive at a more comprehensive picture of researchers’ practices and to reflect the different types of checks required by quantitative analyses. The lenient standard was coded as being met if one or more of the assumptions for a given analytical technique was reported; the stringent standard, by contrast, was recorded as met if all assumptions were reported and checked. Overall, 17% and 24% of the sample were found to have met the stringent and lenient standards, respectively. The study also addresses differences in the checking and reporting of assumptions across the five statistical procedures we examined. Despite overall weak reporting, we note several improvements taking place over time. Suggestions for improving data analytic and reporting practices in the field are discussed as well as implications for researcher training (see Loewen et al., 2014; Plonsky, 2014).

Original languageEnglish (US)
JournalSecond Language Research
DOIs
StateAccepted/In press - Jan 1 2019

Fingerprint

statistical test
language
statistical method
faith
coding
threat
regression
ability
learning
time

Keywords

  • methodological synthesis
  • quantitative research methods
  • second-language research
  • statistical assumptions
  • study quality

ASJC Scopus subject areas

  • Education
  • Linguistics and Language

Cite this

Statistical assumptions in L2 research : A systematic review. / Hu, Yuhang; Plonsky, Luke.

In: Second Language Research, 01.01.2019.

Research output: Contribution to journalReview article

@article{13b85d3d4d4d4690a349ac05d5dd9ccd,
title = "Statistical assumptions in L2 research: A systematic review",
abstract = "Statistical tests carry with them a number of assumptions that must be checked. Failing to do so and to report the results of such preliminary analyses introduce a potential threat to the internal validity of a study and to our ability as consumers to put faith in study findings. This article systematically examines the reporting of checks on assumptions in two major journals of second-language (L2) research (Language Learning and Second Language Research) over a span of six years (2012–2017 inclusive). Using synthetic techniques, each study in the sample (K = 107) was coded for whether or not the assumptions associated with five statistical tests were reported: independent samples t-test, one-way ANOVA, Pearson correlation, multiple regression, and chi-square. In doing so, two coding standards (‘stringent’ and ‘lenient’) were adopted to arrive at a more comprehensive picture of researchers’ practices and to reflect the different types of checks required by quantitative analyses. The lenient standard was coded as being met if one or more of the assumptions for a given analytical technique was reported; the stringent standard, by contrast, was recorded as met if all assumptions were reported and checked. Overall, 17{\%} and 24{\%} of the sample were found to have met the stringent and lenient standards, respectively. The study also addresses differences in the checking and reporting of assumptions across the five statistical procedures we examined. Despite overall weak reporting, we note several improvements taking place over time. Suggestions for improving data analytic and reporting practices in the field are discussed as well as implications for researcher training (see Loewen et al., 2014; Plonsky, 2014).",
keywords = "methodological synthesis, quantitative research methods, second-language research, statistical assumptions, study quality",
author = "Yuhang Hu and Luke Plonsky",
year = "2019",
month = "1",
day = "1",
doi = "10.1177/0267658319877433",
language = "English (US)",
journal = "Second Language Research",
issn = "0267-6583",
publisher = "SAGE Publications Ltd",

}

TY - JOUR

T1 - Statistical assumptions in L2 research

T2 - A systematic review

AU - Hu, Yuhang

AU - Plonsky, Luke

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Statistical tests carry with them a number of assumptions that must be checked. Failing to do so and to report the results of such preliminary analyses introduce a potential threat to the internal validity of a study and to our ability as consumers to put faith in study findings. This article systematically examines the reporting of checks on assumptions in two major journals of second-language (L2) research (Language Learning and Second Language Research) over a span of six years (2012–2017 inclusive). Using synthetic techniques, each study in the sample (K = 107) was coded for whether or not the assumptions associated with five statistical tests were reported: independent samples t-test, one-way ANOVA, Pearson correlation, multiple regression, and chi-square. In doing so, two coding standards (‘stringent’ and ‘lenient’) were adopted to arrive at a more comprehensive picture of researchers’ practices and to reflect the different types of checks required by quantitative analyses. The lenient standard was coded as being met if one or more of the assumptions for a given analytical technique was reported; the stringent standard, by contrast, was recorded as met if all assumptions were reported and checked. Overall, 17% and 24% of the sample were found to have met the stringent and lenient standards, respectively. The study also addresses differences in the checking and reporting of assumptions across the five statistical procedures we examined. Despite overall weak reporting, we note several improvements taking place over time. Suggestions for improving data analytic and reporting practices in the field are discussed as well as implications for researcher training (see Loewen et al., 2014; Plonsky, 2014).

AB - Statistical tests carry with them a number of assumptions that must be checked. Failing to do so and to report the results of such preliminary analyses introduce a potential threat to the internal validity of a study and to our ability as consumers to put faith in study findings. This article systematically examines the reporting of checks on assumptions in two major journals of second-language (L2) research (Language Learning and Second Language Research) over a span of six years (2012–2017 inclusive). Using synthetic techniques, each study in the sample (K = 107) was coded for whether or not the assumptions associated with five statistical tests were reported: independent samples t-test, one-way ANOVA, Pearson correlation, multiple regression, and chi-square. In doing so, two coding standards (‘stringent’ and ‘lenient’) were adopted to arrive at a more comprehensive picture of researchers’ practices and to reflect the different types of checks required by quantitative analyses. The lenient standard was coded as being met if one or more of the assumptions for a given analytical technique was reported; the stringent standard, by contrast, was recorded as met if all assumptions were reported and checked. Overall, 17% and 24% of the sample were found to have met the stringent and lenient standards, respectively. The study also addresses differences in the checking and reporting of assumptions across the five statistical procedures we examined. Despite overall weak reporting, we note several improvements taking place over time. Suggestions for improving data analytic and reporting practices in the field are discussed as well as implications for researcher training (see Loewen et al., 2014; Plonsky, 2014).

KW - methodological synthesis

KW - quantitative research methods

KW - second-language research

KW - statistical assumptions

KW - study quality

UR - http://www.scopus.com/inward/record.url?scp=85074282312&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85074282312&partnerID=8YFLogxK

U2 - 10.1177/0267658319877433

DO - 10.1177/0267658319877433

M3 - Review article

AN - SCOPUS:85074282312

JO - Second Language Research

JF - Second Language Research

SN - 0267-6583

ER -