Sensitivity of inferred climate model skill to evaluation decisions

A case study using CMIP5 evapotranspiration

Christopher R Schwalm, Deborah N. Huntinzger, Anna M. Michalak, Joshua B. Fisher, John S. Kimball, Brigitte Mueller, Ke Zhang, Yongqiang Zhang

Research output: Contribution to journalArticle

20 Citations (Scopus)

Abstract

Confrontation of climate models with observationally-based reference datasets is widespread and integral to model development. These comparisons yield skill metrics quantifying the mismatch between simulated and reference values and also involve analyst choices, or meta-parameters, in structuring the analysis. Here, we systematically vary five such meta-parameters (reference dataset, spatial resolution, regridding approach, land mask, and time period) in evaluating evapotranspiration (ET) from eight CMIP5 models in a factorial design that yields 68 700 intercomparisons. The results show that while model-data comparisons can provide some feedback on overall model performance, model ranks are ambiguous and inferred model skill and rank are highly sensitive to the choice of meta-parameters for all models. This suggests that model skill and rank are best represented probabilistically rather than as scalar values. For this case study, the choice of reference dataset is found to have a dominant influence on inferred model skill, even larger than the choice of model itself. This is primarily due to large differences between reference datasets, indicating that further work in developing a community-accepted standard ET reference dataset is crucial in order to decrease ambiguity in model skill.

Original languageEnglish (US)
Article number024028
JournalEnvironmental Research Letters
Volume8
Issue number2
DOIs
StatePublished - 2013

Fingerprint

Climate models
Evapotranspiration
Climate
evapotranspiration
climate modeling
Masks
Reference Values
decision
CMIP
evaluation
Datasets
spatial resolution

Keywords

  • climate models
  • CMIP5
  • evapotranspiration
  • model validation

ASJC Scopus subject areas

  • Environmental Science(all)
  • Renewable Energy, Sustainability and the Environment
  • Public Health, Environmental and Occupational Health

Cite this

Sensitivity of inferred climate model skill to evaluation decisions : A case study using CMIP5 evapotranspiration. / Schwalm, Christopher R; Huntinzger, Deborah N.; Michalak, Anna M.; Fisher, Joshua B.; Kimball, John S.; Mueller, Brigitte; Zhang, Ke; Zhang, Yongqiang.

In: Environmental Research Letters, Vol. 8, No. 2, 024028, 2013.

Research output: Contribution to journalArticle

Schwalm, Christopher R ; Huntinzger, Deborah N. ; Michalak, Anna M. ; Fisher, Joshua B. ; Kimball, John S. ; Mueller, Brigitte ; Zhang, Ke ; Zhang, Yongqiang. / Sensitivity of inferred climate model skill to evaluation decisions : A case study using CMIP5 evapotranspiration. In: Environmental Research Letters. 2013 ; Vol. 8, No. 2.
@article{5eca3c0e0b4e40089efc41f723506339,
title = "Sensitivity of inferred climate model skill to evaluation decisions: A case study using CMIP5 evapotranspiration",
abstract = "Confrontation of climate models with observationally-based reference datasets is widespread and integral to model development. These comparisons yield skill metrics quantifying the mismatch between simulated and reference values and also involve analyst choices, or meta-parameters, in structuring the analysis. Here, we systematically vary five such meta-parameters (reference dataset, spatial resolution, regridding approach, land mask, and time period) in evaluating evapotranspiration (ET) from eight CMIP5 models in a factorial design that yields 68 700 intercomparisons. The results show that while model-data comparisons can provide some feedback on overall model performance, model ranks are ambiguous and inferred model skill and rank are highly sensitive to the choice of meta-parameters for all models. This suggests that model skill and rank are best represented probabilistically rather than as scalar values. For this case study, the choice of reference dataset is found to have a dominant influence on inferred model skill, even larger than the choice of model itself. This is primarily due to large differences between reference datasets, indicating that further work in developing a community-accepted standard ET reference dataset is crucial in order to decrease ambiguity in model skill.",
keywords = "climate models, CMIP5, evapotranspiration, model validation",
author = "Schwalm, {Christopher R} and Huntinzger, {Deborah N.} and Michalak, {Anna M.} and Fisher, {Joshua B.} and Kimball, {John S.} and Brigitte Mueller and Ke Zhang and Yongqiang Zhang",
year = "2013",
doi = "10.1088/1748-9326/8/2/024028",
language = "English (US)",
volume = "8",
journal = "Environmental Research Letters",
issn = "1748-9326",
publisher = "IOP Publishing Ltd.",
number = "2",

}

TY - JOUR

T1 - Sensitivity of inferred climate model skill to evaluation decisions

T2 - A case study using CMIP5 evapotranspiration

AU - Schwalm, Christopher R

AU - Huntinzger, Deborah N.

AU - Michalak, Anna M.

AU - Fisher, Joshua B.

AU - Kimball, John S.

AU - Mueller, Brigitte

AU - Zhang, Ke

AU - Zhang, Yongqiang

PY - 2013

Y1 - 2013

N2 - Confrontation of climate models with observationally-based reference datasets is widespread and integral to model development. These comparisons yield skill metrics quantifying the mismatch between simulated and reference values and also involve analyst choices, or meta-parameters, in structuring the analysis. Here, we systematically vary five such meta-parameters (reference dataset, spatial resolution, regridding approach, land mask, and time period) in evaluating evapotranspiration (ET) from eight CMIP5 models in a factorial design that yields 68 700 intercomparisons. The results show that while model-data comparisons can provide some feedback on overall model performance, model ranks are ambiguous and inferred model skill and rank are highly sensitive to the choice of meta-parameters for all models. This suggests that model skill and rank are best represented probabilistically rather than as scalar values. For this case study, the choice of reference dataset is found to have a dominant influence on inferred model skill, even larger than the choice of model itself. This is primarily due to large differences between reference datasets, indicating that further work in developing a community-accepted standard ET reference dataset is crucial in order to decrease ambiguity in model skill.

AB - Confrontation of climate models with observationally-based reference datasets is widespread and integral to model development. These comparisons yield skill metrics quantifying the mismatch between simulated and reference values and also involve analyst choices, or meta-parameters, in structuring the analysis. Here, we systematically vary five such meta-parameters (reference dataset, spatial resolution, regridding approach, land mask, and time period) in evaluating evapotranspiration (ET) from eight CMIP5 models in a factorial design that yields 68 700 intercomparisons. The results show that while model-data comparisons can provide some feedback on overall model performance, model ranks are ambiguous and inferred model skill and rank are highly sensitive to the choice of meta-parameters for all models. This suggests that model skill and rank are best represented probabilistically rather than as scalar values. For this case study, the choice of reference dataset is found to have a dominant influence on inferred model skill, even larger than the choice of model itself. This is primarily due to large differences between reference datasets, indicating that further work in developing a community-accepted standard ET reference dataset is crucial in order to decrease ambiguity in model skill.

KW - climate models

KW - CMIP5

KW - evapotranspiration

KW - model validation

UR - http://www.scopus.com/inward/record.url?scp=84880882512&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84880882512&partnerID=8YFLogxK

U2 - 10.1088/1748-9326/8/2/024028

DO - 10.1088/1748-9326/8/2/024028

M3 - Article

VL - 8

JO - Environmental Research Letters

JF - Environmental Research Letters

SN - 1748-9326

IS - 2

M1 - 024028

ER -