Study quality in quantitative l2 research (1990-2010): A methodological synthesis and call for reform

Research output: Contribution to journalArticle

56 Citations (Scopus)

Abstract

This article builds on the growing line of inquiry into methodological practices in quantitative second language (L2) research. Specifically, the study uses synthetic techniques to examine changes over time in research and reporting practices. 606 primary reports of quantitative L2 research from two journals-Language Learning and Studies in Second Language Acquisition-were surveyed on different design features, statistical analyses, and data reporting practices. Frequencies and percentages of each feature were then calculated and compared across the 1990s and the first decade of the 2000s to examine changes taking place in the field. The results, while not necessarily representative of all substantive and methodological approaches within the domain of L2 research, indicate numerous changes including increases in sample sizes, delayed posttesting, and the availability of critical data such as effect sizes, reliability estimates, and standard deviations to accompany means. With respect to statistical procedures, the range of analyses has not changed, and the field continues its unfortunate reliance on statistical significance. The findings are grouped according to three themes, which are discussed in light of previous reviews in this and other fields: (a) means-based analyses, (b) missing data, null hypothesis significance testing, and the "power problem," and (c) design preferences. The article concludes with an extended call for reform targeting six groups of stakeholders in the field. Most notably, an argument is made for field-specific methodological standards and enhancements to graduate curricula and training.

Original languageEnglish (US)
Pages (from-to)450-470
Number of pages21
JournalModern Language Journal
Volume98
Issue number1
DOIs
StatePublished - 2014

Fingerprint

quantitative research
reform
statistical significance
language
language acquisition
statistical method
stakeholder
graduate
curriculum
learning
Group

Keywords

  • Meta-analysis
  • Quantitative research methods
  • Research synthesis
  • Statistical reform
  • Study quality

ASJC Scopus subject areas

  • Linguistics and Language
  • Language and Linguistics

Cite this

Study quality in quantitative l2 research (1990-2010) : A methodological synthesis and call for reform. / Plonsky, Luke D.

In: Modern Language Journal, Vol. 98, No. 1, 2014, p. 450-470.

Research output: Contribution to journalArticle

@article{67f2b7ed5e614893988ef7e5dea56c1f,
title = "Study quality in quantitative l2 research (1990-2010): A methodological synthesis and call for reform",
abstract = "This article builds on the growing line of inquiry into methodological practices in quantitative second language (L2) research. Specifically, the study uses synthetic techniques to examine changes over time in research and reporting practices. 606 primary reports of quantitative L2 research from two journals-Language Learning and Studies in Second Language Acquisition-were surveyed on different design features, statistical analyses, and data reporting practices. Frequencies and percentages of each feature were then calculated and compared across the 1990s and the first decade of the 2000s to examine changes taking place in the field. The results, while not necessarily representative of all substantive and methodological approaches within the domain of L2 research, indicate numerous changes including increases in sample sizes, delayed posttesting, and the availability of critical data such as effect sizes, reliability estimates, and standard deviations to accompany means. With respect to statistical procedures, the range of analyses has not changed, and the field continues its unfortunate reliance on statistical significance. The findings are grouped according to three themes, which are discussed in light of previous reviews in this and other fields: (a) means-based analyses, (b) missing data, null hypothesis significance testing, and the {"}power problem,{"} and (c) design preferences. The article concludes with an extended call for reform targeting six groups of stakeholders in the field. Most notably, an argument is made for field-specific methodological standards and enhancements to graduate curricula and training.",
keywords = "Meta-analysis, Quantitative research methods, Research synthesis, Statistical reform, Study quality",
author = "Plonsky, {Luke D}",
year = "2014",
doi = "10.1111/j.1540-4781.2014.12058.x",
language = "English (US)",
volume = "98",
pages = "450--470",
journal = "Modern Language Journal",
issn = "0026-7902",
publisher = "Wiley-Blackwell",
number = "1",

}

TY - JOUR

T1 - Study quality in quantitative l2 research (1990-2010)

T2 - A methodological synthesis and call for reform

AU - Plonsky, Luke D

PY - 2014

Y1 - 2014

N2 - This article builds on the growing line of inquiry into methodological practices in quantitative second language (L2) research. Specifically, the study uses synthetic techniques to examine changes over time in research and reporting practices. 606 primary reports of quantitative L2 research from two journals-Language Learning and Studies in Second Language Acquisition-were surveyed on different design features, statistical analyses, and data reporting practices. Frequencies and percentages of each feature were then calculated and compared across the 1990s and the first decade of the 2000s to examine changes taking place in the field. The results, while not necessarily representative of all substantive and methodological approaches within the domain of L2 research, indicate numerous changes including increases in sample sizes, delayed posttesting, and the availability of critical data such as effect sizes, reliability estimates, and standard deviations to accompany means. With respect to statistical procedures, the range of analyses has not changed, and the field continues its unfortunate reliance on statistical significance. The findings are grouped according to three themes, which are discussed in light of previous reviews in this and other fields: (a) means-based analyses, (b) missing data, null hypothesis significance testing, and the "power problem," and (c) design preferences. The article concludes with an extended call for reform targeting six groups of stakeholders in the field. Most notably, an argument is made for field-specific methodological standards and enhancements to graduate curricula and training.

AB - This article builds on the growing line of inquiry into methodological practices in quantitative second language (L2) research. Specifically, the study uses synthetic techniques to examine changes over time in research and reporting practices. 606 primary reports of quantitative L2 research from two journals-Language Learning and Studies in Second Language Acquisition-were surveyed on different design features, statistical analyses, and data reporting practices. Frequencies and percentages of each feature were then calculated and compared across the 1990s and the first decade of the 2000s to examine changes taking place in the field. The results, while not necessarily representative of all substantive and methodological approaches within the domain of L2 research, indicate numerous changes including increases in sample sizes, delayed posttesting, and the availability of critical data such as effect sizes, reliability estimates, and standard deviations to accompany means. With respect to statistical procedures, the range of analyses has not changed, and the field continues its unfortunate reliance on statistical significance. The findings are grouped according to three themes, which are discussed in light of previous reviews in this and other fields: (a) means-based analyses, (b) missing data, null hypothesis significance testing, and the "power problem," and (c) design preferences. The article concludes with an extended call for reform targeting six groups of stakeholders in the field. Most notably, an argument is made for field-specific methodological standards and enhancements to graduate curricula and training.

KW - Meta-analysis

KW - Quantitative research methods

KW - Research synthesis

KW - Statistical reform

KW - Study quality

UR - http://www.scopus.com/inward/record.url?scp=84894107678&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84894107678&partnerID=8YFLogxK

U2 - 10.1111/j.1540-4781.2014.12058.x

DO - 10.1111/j.1540-4781.2014.12058.x

M3 - Article

AN - SCOPUS:84894107678

VL - 98

SP - 450

EP - 470

JO - Modern Language Journal

JF - Modern Language Journal

SN - 0026-7902

IS - 1

ER -