Intrinsic evaluation of text mining tools may not predict performance on realistic tasks

J. Gregory Caporaso, Nita Deshpande, J. Lynn Fink, Philip E. Bourne, K. Bretonnel Cohen, Lawrence Hunter

Research output: Chapter in Book/Report/Conference proceedingConference contribution

30 Scopus citations


Biomedical text mining and other automated techniques are beginning to achieve performance which suggests that they could be applied to aid database curators. However, few studies have evaluated how these systems might work in practice. In this article we focus on the problem of annotating mutations in Protein Data Bank (PDB) entries, and evaluate the relationship between performance of two automated techniques, a text-mining-based approach (MutationFinder) and an alignment-based approach, in intrinsic versus extrinsic evaluations. We find that high performance on gold standard data (an intrinsic evaluation) does not necessarily translate to high performance for database annotation (an extrinsic evaluation). We show that this is in part a result of lack of access to the full text of journal articles, which appears to be critical for comprehensive database annotation by text mining. Additionally, we evaluate the accuracy and completeness of manually annotated mutation data in the PDB, and find that it is far from perfect. We conclude that currently the most cost-effective and reliable approach for database annotation might incorporate manual and automatic annotation methods.

Original languageEnglish (US)
Title of host publicationPacific Symposium on Biocomputing 2008, PSB 2008
Number of pages12
StatePublished - 2008
Externally publishedYes
Event13th Pacific Symposium on Biocomputing, PSB 2008 - Kohala Coast, HI, United States
Duration: Jan 4 2008Jan 8 2008

Publication series

NamePacific Symposium on Biocomputing 2008, PSB 2008


Other13th Pacific Symposium on Biocomputing, PSB 2008
Country/TerritoryUnited States
CityKohala Coast, HI

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Biomedical Engineering
  • Medicine(all)


Dive into the research topics of 'Intrinsic evaluation of text mining tools may not predict performance on realistic tasks'. Together they form a unique fingerprint.

Cite this