The effect of training and rater differences on oral proficiency assessment

Okim Kang, Don Rubin, Alyssa Kermad

Research output: Contribution to journalArticle

Abstract

As a result of the fact that judgments of non-native speech are closely tied to social biases, oral proficiency ratings are susceptible to error because of rater background and social attitudes. The present study seeks first to estimate the variance attributable to rater background and attitudinal variables on novice raters’ assessments of L2 spoken English. Second, the study examines the effects of minimal training in reducing the potency of those trait-irrelevant rater factors. Accordingly, this study examined the relative impact of rater differences on TOEFL iBT® speaking scores. Eighty-two untrained raters judged 112 speech samples produced by TOEFL® examinees. Findings revealed that approximately 20% of untrained raters’ score variance was, in part, a result of their background and attitudinal factors. The strongest predictor was the raters’ own native speaker status. However, minimal online training dramatically reduced the impact of rater background and attitudinal variables for a subsample of high- and low-severity raters. Implications suggest that brief and user-friendly rater-training sessions offer the promise of mitigating rater bias, at least in the short run. This procedure can be adopted in assessment and other related fields of applied linguistics.

Original languageEnglish (US)
JournalLanguage Testing
DOIs
StatePublished - Jan 1 2019

Fingerprint

social attitude
trend
speaking
rating
linguistics
Raters
Proficiency
TOEFL

Keywords

  • Oral proficiency
  • rater background
  • rater training
  • speaking assessment
  • stereotypes of non-native speakers

ASJC Scopus subject areas

  • Language and Linguistics
  • Social Sciences (miscellaneous)
  • Linguistics and Language

Cite this

The effect of training and rater differences on oral proficiency assessment. / Kang, Okim; Rubin, Don; Kermad, Alyssa.

In: Language Testing, 01.01.2019.

Research output: Contribution to journalArticle

@article{a59a65c3a77c4ab58f7063146dbb93dd,
title = "The effect of training and rater differences on oral proficiency assessment",
abstract = "As a result of the fact that judgments of non-native speech are closely tied to social biases, oral proficiency ratings are susceptible to error because of rater background and social attitudes. The present study seeks first to estimate the variance attributable to rater background and attitudinal variables on novice raters’ assessments of L2 spoken English. Second, the study examines the effects of minimal training in reducing the potency of those trait-irrelevant rater factors. Accordingly, this study examined the relative impact of rater differences on TOEFL iBT{\circledR} speaking scores. Eighty-two untrained raters judged 112 speech samples produced by TOEFL{\circledR} examinees. Findings revealed that approximately 20{\%} of untrained raters’ score variance was, in part, a result of their background and attitudinal factors. The strongest predictor was the raters’ own native speaker status. However, minimal online training dramatically reduced the impact of rater background and attitudinal variables for a subsample of high- and low-severity raters. Implications suggest that brief and user-friendly rater-training sessions offer the promise of mitigating rater bias, at least in the short run. This procedure can be adopted in assessment and other related fields of applied linguistics.",
keywords = "Oral proficiency, rater background, rater training, speaking assessment, stereotypes of non-native speakers",
author = "Okim Kang and Don Rubin and Alyssa Kermad",
year = "2019",
month = "1",
day = "1",
doi = "10.1177/0265532219849522",
language = "English (US)",
journal = "Language Testing",
issn = "0265-5322",
publisher = "SAGE Publications Ltd",

}

TY - JOUR

T1 - The effect of training and rater differences on oral proficiency assessment

AU - Kang, Okim

AU - Rubin, Don

AU - Kermad, Alyssa

PY - 2019/1/1

Y1 - 2019/1/1

N2 - As a result of the fact that judgments of non-native speech are closely tied to social biases, oral proficiency ratings are susceptible to error because of rater background and social attitudes. The present study seeks first to estimate the variance attributable to rater background and attitudinal variables on novice raters’ assessments of L2 spoken English. Second, the study examines the effects of minimal training in reducing the potency of those trait-irrelevant rater factors. Accordingly, this study examined the relative impact of rater differences on TOEFL iBT® speaking scores. Eighty-two untrained raters judged 112 speech samples produced by TOEFL® examinees. Findings revealed that approximately 20% of untrained raters’ score variance was, in part, a result of their background and attitudinal factors. The strongest predictor was the raters’ own native speaker status. However, minimal online training dramatically reduced the impact of rater background and attitudinal variables for a subsample of high- and low-severity raters. Implications suggest that brief and user-friendly rater-training sessions offer the promise of mitigating rater bias, at least in the short run. This procedure can be adopted in assessment and other related fields of applied linguistics.

AB - As a result of the fact that judgments of non-native speech are closely tied to social biases, oral proficiency ratings are susceptible to error because of rater background and social attitudes. The present study seeks first to estimate the variance attributable to rater background and attitudinal variables on novice raters’ assessments of L2 spoken English. Second, the study examines the effects of minimal training in reducing the potency of those trait-irrelevant rater factors. Accordingly, this study examined the relative impact of rater differences on TOEFL iBT® speaking scores. Eighty-two untrained raters judged 112 speech samples produced by TOEFL® examinees. Findings revealed that approximately 20% of untrained raters’ score variance was, in part, a result of their background and attitudinal factors. The strongest predictor was the raters’ own native speaker status. However, minimal online training dramatically reduced the impact of rater background and attitudinal variables for a subsample of high- and low-severity raters. Implications suggest that brief and user-friendly rater-training sessions offer the promise of mitigating rater bias, at least in the short run. This procedure can be adopted in assessment and other related fields of applied linguistics.

KW - Oral proficiency

KW - rater background

KW - rater training

KW - speaking assessment

KW - stereotypes of non-native speakers

UR - http://www.scopus.com/inward/record.url?scp=85068003852&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85068003852&partnerID=8YFLogxK

U2 - 10.1177/0265532219849522

DO - 10.1177/0265532219849522

M3 - Article

AN - SCOPUS:85068003852

JO - Language Testing

JF - Language Testing

SN - 0265-5322

ER -