Using Corpus-Based Register Analysis to Explore the Authenticity of High-Stakes Language Exams

A Register Comparison of TOEFL iBT and Disciplinary Writing Tasks

Shelley Staples, Douglas E Biber, Randi Reppen

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

One of the central considerations in the validity argument for the TOEFL iBT is the relationship between the language on the exam and the language required for university courses. Corpus linguistics has recently been shown to be an effective way to explore this relationship, which can also be considered as an aspect of authenticity. Applying Multi-Dimensional Analysis, our study is the first to directly compare the lexico-grammatical characteristics of texts produced by second language writers on a high-stakes standardized exam (the TOEFL iBT) to the characteristics of texts produced by the same writers for disciplinary writing tasks required as part of their academic career (e.g., lab reports, argumentative essays). Our findings show that the language of TOEFL iBT tasks have patterns of both similarity and difference from disciplinary tasks with respect to four linguistic dimensions of language use: (a) Compressed Procedural Information versus Stance toward the Work of Others, (b) Personal Stance, (c) Possible versus Completed Events, and (d) Information Density. Integrated iBT tasks show similarities to certain kinds of disciplinary tasks, while independent iBT tasks differ linguistically from almost all disciplinary writing tasks. These general patterns hold regardless of the score levels of test takers. We discuss the implications of these findings for language teaching and assessment.

Original languageEnglish (US)
JournalModern Language Journal
DOIs
StateAccepted/In press - Jan 1 2018

Fingerprint

authenticity
language
writer
linguistics
dimensional analysis
academic career
Authenticity
High-stakes
TOEFL
Corpus-based
Language
Disciplinary Writing
event
university
Teaching
Stance
Writer

Keywords

  • Assessment
  • Corpus linguistics
  • English for Academic Purposes (EAP)
  • Research methodology
  • Writing

ASJC Scopus subject areas

  • Language and Linguistics
  • Linguistics and Language

Cite this

@article{2ed44d52d8574444b61479dd7b7261c0,
title = "Using Corpus-Based Register Analysis to Explore the Authenticity of High-Stakes Language Exams: A Register Comparison of TOEFL iBT and Disciplinary Writing Tasks",
abstract = "One of the central considerations in the validity argument for the TOEFL iBT is the relationship between the language on the exam and the language required for university courses. Corpus linguistics has recently been shown to be an effective way to explore this relationship, which can also be considered as an aspect of authenticity. Applying Multi-Dimensional Analysis, our study is the first to directly compare the lexico-grammatical characteristics of texts produced by second language writers on a high-stakes standardized exam (the TOEFL iBT) to the characteristics of texts produced by the same writers for disciplinary writing tasks required as part of their academic career (e.g., lab reports, argumentative essays). Our findings show that the language of TOEFL iBT tasks have patterns of both similarity and difference from disciplinary tasks with respect to four linguistic dimensions of language use: (a) Compressed Procedural Information versus Stance toward the Work of Others, (b) Personal Stance, (c) Possible versus Completed Events, and (d) Information Density. Integrated iBT tasks show similarities to certain kinds of disciplinary tasks, while independent iBT tasks differ linguistically from almost all disciplinary writing tasks. These general patterns hold regardless of the score levels of test takers. We discuss the implications of these findings for language teaching and assessment.",
keywords = "Assessment, Corpus linguistics, English for Academic Purposes (EAP), Research methodology, Writing",
author = "Shelley Staples and Biber, {Douglas E} and Randi Reppen",
year = "2018",
month = "1",
day = "1",
doi = "10.1111/modl.12465",
language = "English (US)",
journal = "Modern Language Journal",
issn = "0026-7902",
publisher = "Wiley-Blackwell",

}

TY - JOUR

T1 - Using Corpus-Based Register Analysis to Explore the Authenticity of High-Stakes Language Exams

T2 - A Register Comparison of TOEFL iBT and Disciplinary Writing Tasks

AU - Staples, Shelley

AU - Biber, Douglas E

AU - Reppen, Randi

PY - 2018/1/1

Y1 - 2018/1/1

N2 - One of the central considerations in the validity argument for the TOEFL iBT is the relationship between the language on the exam and the language required for university courses. Corpus linguistics has recently been shown to be an effective way to explore this relationship, which can also be considered as an aspect of authenticity. Applying Multi-Dimensional Analysis, our study is the first to directly compare the lexico-grammatical characteristics of texts produced by second language writers on a high-stakes standardized exam (the TOEFL iBT) to the characteristics of texts produced by the same writers for disciplinary writing tasks required as part of their academic career (e.g., lab reports, argumentative essays). Our findings show that the language of TOEFL iBT tasks have patterns of both similarity and difference from disciplinary tasks with respect to four linguistic dimensions of language use: (a) Compressed Procedural Information versus Stance toward the Work of Others, (b) Personal Stance, (c) Possible versus Completed Events, and (d) Information Density. Integrated iBT tasks show similarities to certain kinds of disciplinary tasks, while independent iBT tasks differ linguistically from almost all disciplinary writing tasks. These general patterns hold regardless of the score levels of test takers. We discuss the implications of these findings for language teaching and assessment.

AB - One of the central considerations in the validity argument for the TOEFL iBT is the relationship between the language on the exam and the language required for university courses. Corpus linguistics has recently been shown to be an effective way to explore this relationship, which can also be considered as an aspect of authenticity. Applying Multi-Dimensional Analysis, our study is the first to directly compare the lexico-grammatical characteristics of texts produced by second language writers on a high-stakes standardized exam (the TOEFL iBT) to the characteristics of texts produced by the same writers for disciplinary writing tasks required as part of their academic career (e.g., lab reports, argumentative essays). Our findings show that the language of TOEFL iBT tasks have patterns of both similarity and difference from disciplinary tasks with respect to four linguistic dimensions of language use: (a) Compressed Procedural Information versus Stance toward the Work of Others, (b) Personal Stance, (c) Possible versus Completed Events, and (d) Information Density. Integrated iBT tasks show similarities to certain kinds of disciplinary tasks, while independent iBT tasks differ linguistically from almost all disciplinary writing tasks. These general patterns hold regardless of the score levels of test takers. We discuss the implications of these findings for language teaching and assessment.

KW - Assessment

KW - Corpus linguistics

KW - English for Academic Purposes (EAP)

KW - Research methodology

KW - Writing

UR - http://www.scopus.com/inward/record.url?scp=85042169515&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85042169515&partnerID=8YFLogxK

U2 - 10.1111/modl.12465

DO - 10.1111/modl.12465

M3 - Article

JO - Modern Language Journal

JF - Modern Language Journal

SN - 0026-7902

ER -