Domain definition and search techniques in meta-analyses of L2 research (Or why 18 meta-analyses of feedback have different results)

Luke D Plonsky, Dan Brown

Research output: Contribution to journalArticle

26 Citations (Scopus)

Abstract

Applied linguists have turned increasingly in recent years to meta-analysis as the preferred means of synthesizing quantitative research. The first step in the meta-analytic process involves defining a domain of interest. Despite its apparent simplicity, this step involves a great deal of subjectivity on the part of the meta-analyst. This article problematizes the importance of clearly defining and operationalizing meta-analytic domains. Toward that end, we present a critical review of one particular domain, corrective feedback, which has been subject to 18 unique meta-analyses. Specifically, we examine the unique approach each study has taken in defining their domain of interest. In order to demonstrate the critical role of this stage in the meta-analytic process, we also examine variability in summary effects as a function of the unique subdomains in the sample. Because different techniques used to identify candidate studies carry assumptions about the type of research that falls within the domain of interest (e.g. published vs. unpublished), we also include a brief review of search techniques employed in a set of 81 meta-analyses of second language research. Building on the work of In’nami and Koizumi (2010) and Oswald and Plonsky (2010), the results for this phase of the analysis show that L2 meta-analysts generally rely on a stable but very limited set of search strategies, none of which is likely to yield unpublished studies. Based on our findings related both to domain definition and search techniques employed by L2 researchers, we make specific recommendations for future meta-analytic practice in the field.

Original languageEnglish (US)
Pages (from-to)267-278
Number of pages12
JournalSecond Language Research
Volume31
Issue number2
DOIs
StatePublished - Apr 20 2015

Fingerprint

quantitative research
subjectivity
candidacy
language

Keywords

  • corrective feedback
  • domain definition
  • meta-analysis
  • search techniques

ASJC Scopus subject areas

  • Linguistics and Language
  • Education

Cite this

@article{5a9eb028e7614b6bbbad93dec14fe06e,
title = "Domain definition and search techniques in meta-analyses of L2 research (Or why 18 meta-analyses of feedback have different results)",
abstract = "Applied linguists have turned increasingly in recent years to meta-analysis as the preferred means of synthesizing quantitative research. The first step in the meta-analytic process involves defining a domain of interest. Despite its apparent simplicity, this step involves a great deal of subjectivity on the part of the meta-analyst. This article problematizes the importance of clearly defining and operationalizing meta-analytic domains. Toward that end, we present a critical review of one particular domain, corrective feedback, which has been subject to 18 unique meta-analyses. Specifically, we examine the unique approach each study has taken in defining their domain of interest. In order to demonstrate the critical role of this stage in the meta-analytic process, we also examine variability in summary effects as a function of the unique subdomains in the sample. Because different techniques used to identify candidate studies carry assumptions about the type of research that falls within the domain of interest (e.g. published vs. unpublished), we also include a brief review of search techniques employed in a set of 81 meta-analyses of second language research. Building on the work of In’nami and Koizumi (2010) and Oswald and Plonsky (2010), the results for this phase of the analysis show that L2 meta-analysts generally rely on a stable but very limited set of search strategies, none of which is likely to yield unpublished studies. Based on our findings related both to domain definition and search techniques employed by L2 researchers, we make specific recommendations for future meta-analytic practice in the field.",
keywords = "corrective feedback, domain definition, meta-analysis, search techniques",
author = "Plonsky, {Luke D} and Dan Brown",
year = "2015",
month = "4",
day = "20",
doi = "10.1177/0267658314536436",
language = "English (US)",
volume = "31",
pages = "267--278",
journal = "Second Language Research",
issn = "0267-6583",
publisher = "SAGE Publications Ltd",
number = "2",

}

TY - JOUR

T1 - Domain definition and search techniques in meta-analyses of L2 research (Or why 18 meta-analyses of feedback have different results)

AU - Plonsky, Luke D

AU - Brown, Dan

PY - 2015/4/20

Y1 - 2015/4/20

N2 - Applied linguists have turned increasingly in recent years to meta-analysis as the preferred means of synthesizing quantitative research. The first step in the meta-analytic process involves defining a domain of interest. Despite its apparent simplicity, this step involves a great deal of subjectivity on the part of the meta-analyst. This article problematizes the importance of clearly defining and operationalizing meta-analytic domains. Toward that end, we present a critical review of one particular domain, corrective feedback, which has been subject to 18 unique meta-analyses. Specifically, we examine the unique approach each study has taken in defining their domain of interest. In order to demonstrate the critical role of this stage in the meta-analytic process, we also examine variability in summary effects as a function of the unique subdomains in the sample. Because different techniques used to identify candidate studies carry assumptions about the type of research that falls within the domain of interest (e.g. published vs. unpublished), we also include a brief review of search techniques employed in a set of 81 meta-analyses of second language research. Building on the work of In’nami and Koizumi (2010) and Oswald and Plonsky (2010), the results for this phase of the analysis show that L2 meta-analysts generally rely on a stable but very limited set of search strategies, none of which is likely to yield unpublished studies. Based on our findings related both to domain definition and search techniques employed by L2 researchers, we make specific recommendations for future meta-analytic practice in the field.

AB - Applied linguists have turned increasingly in recent years to meta-analysis as the preferred means of synthesizing quantitative research. The first step in the meta-analytic process involves defining a domain of interest. Despite its apparent simplicity, this step involves a great deal of subjectivity on the part of the meta-analyst. This article problematizes the importance of clearly defining and operationalizing meta-analytic domains. Toward that end, we present a critical review of one particular domain, corrective feedback, which has been subject to 18 unique meta-analyses. Specifically, we examine the unique approach each study has taken in defining their domain of interest. In order to demonstrate the critical role of this stage in the meta-analytic process, we also examine variability in summary effects as a function of the unique subdomains in the sample. Because different techniques used to identify candidate studies carry assumptions about the type of research that falls within the domain of interest (e.g. published vs. unpublished), we also include a brief review of search techniques employed in a set of 81 meta-analyses of second language research. Building on the work of In’nami and Koizumi (2010) and Oswald and Plonsky (2010), the results for this phase of the analysis show that L2 meta-analysts generally rely on a stable but very limited set of search strategies, none of which is likely to yield unpublished studies. Based on our findings related both to domain definition and search techniques employed by L2 researchers, we make specific recommendations for future meta-analytic practice in the field.

KW - corrective feedback

KW - domain definition

KW - meta-analysis

KW - search techniques

UR - http://www.scopus.com/inward/record.url?scp=84924933701&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84924933701&partnerID=8YFLogxK

U2 - 10.1177/0267658314536436

DO - 10.1177/0267658314536436

M3 - Article

AN - SCOPUS:84924933701

VL - 31

SP - 267

EP - 278

JO - Second Language Research

JF - Second Language Research

SN - 0267-6583

IS - 2

ER -