Intrinsic evaluation of text mining tools may not predict performance on realistic tasks

J. Gregory Caporaso, Nita Deshpande, J. Lynn Fink, Philip E. Bourne, K. Bretonnel Cohen, Lawrence Hunter

Research output: Chapter in Book/Report/Conference proceedingConference contribution

29 Scopus citations

Abstract

Biomedical text mining and other automated techniques are beginning to achieve performance which suggests that they could be applied to aid database curators. However, few studies have evaluated how these systems might work in practice. In this article we focus on the problem of annotating mutations in Protein Data Bank (PDB) entries, and evaluate the relationship between performance of two automated techniques, a text-mining-based approach (MutationFinder) and an alignment-based approach, in intrinsic versus extrinsic evaluations. We find that high performance on gold standard data (an intrinsic evaluation) does not necessarily translate to high performance for database annotation (an extrinsic evaluation). We show that this is in part a result of lack of access to the full text of journal articles, which appears to be critical for comprehensive database annotation by text mining. Additionally, we evaluate the accuracy and completeness of manually annotated mutation data in the PDB, and find that it is far from perfect. We conclude that currently the most cost-effective and reliable approach for database annotation might incorporate manual and automatic annotation methods.

Original languageEnglish (US)
Title of host publicationPacific Symposium on Biocomputing 2008, PSB 2008
Pages640-651
Number of pages12
StatePublished - 2008
Event13th Pacific Symposium on Biocomputing, PSB 2008 - Kohala Coast, HI, United States
Duration: Jan 4 2008Jan 8 2008

Publication series

NamePacific Symposium on Biocomputing 2008, PSB 2008

Other

Other13th Pacific Symposium on Biocomputing, PSB 2008
CountryUnited States
CityKohala Coast, HI
Period1/4/081/8/08

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Biomedical Engineering
  • Medicine(all)

Fingerprint Dive into the research topics of 'Intrinsic evaluation of text mining tools may not predict performance on realistic tasks'. Together they form a unique fingerprint.

  • Cite this

    Caporaso, J. G., Deshpande, N., Fink, J. L., Bourne, P. E., Cohen, K. B., & Hunter, L. (2008). Intrinsic evaluation of text mining tools may not predict performance on realistic tasks. In Pacific Symposium on Biocomputing 2008, PSB 2008 (pp. 640-651). (Pacific Symposium on Biocomputing 2008, PSB 2008).