Crowdsourcing consensus: proposal of a novel method for assessing accuracy in echocardiography interpretation

Stephanie Minter, Alicia Armour, Amanda Tinnemore, Karen Strub, Anna Lisa Crowley, Gerald S. Bloomfield, John H. Alexander, Pamela S. Douglas, Joseph A. Kisslo, Eric J. Velazquez, Zainab Samad

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)


Quality in stress echocardiography interpretation is often gauged against coronary angiography (CA) data but anatomic obstructive coronary disease on CA is an imperfect gold standard for a stress induced wall motion abnormality. We examined the utility of crowd-sourcing a “majority-vote” consensus as an alternative ‘gold standard’ against which to evaluate the accuracy of an individual echocardiographer’s interpretation of stress echocardiography studies. Participants independently interpreted baseline and post-exercise stress echocardiographic images of cases that had undergone follow up CA within 3 months of the stress echo in two surveys, 2 years apart. We examined the agreement of consensus on survey (survey participant response (> 60%) for one decision) with the stress echocardiography clinical read and with CA results. In the first survey, 29 participants reviewed and independently interpreted 14 stress echo cases. Consensus was reached in all 14 cases. There was good agreement between clinical and consensus (kappa = 0.57), survey participant response and consensus (kappa = 0.68) and consensus and CA results (kappa = 0.40). In the validation survey, the agreement between clinical reads and consensus (kappa = 0.75) and survey participant response and consensus (kappa = 0.81) remained excellent. Independent consensus is achievable and offers a fair comparison for stress echocardiographic interpretation. Future validation work, in other laboratories, and against hard outcomes, is necessary to test the feasibility and effectiveness of this approach.

Original languageEnglish
Pages (from-to)1725-1730
Number of pages6
JournalInternational Journal of Cardiovascular Imaging
Issue number11
Publication statusPublished - 1 Nov 2018
Externally publishedYes


  • Cardiac catheterization
  • Continuous quality improvement
  • Echocardiography
  • Image acquisition
  • Stress echocardiography


Dive into the research topics of 'Crowdsourcing consensus: proposal of a novel method for assessing accuracy in echocardiography interpretation'. Together they form a unique fingerprint.

Cite this