Inter-observer Reliability for Chest X-Ray Interpretation in COVID-19 Patients of Radiologists with Different Experiences

Authors

  • Wichunun Kunchai Radiologist, Chaiyaphum Hospital, Chaiyaphum Province.

Keywords:

Interobserver reliability, COVID-19 CXR, COVID pneumonia severity score

Abstract

Background and Objectives: Radiographs play a very important role in diagnosing and planning patient care. This study aimed to assess the inter- and intraobserver reliability among radiologists, particularly between new and experienced radiologists, in evaluating the extent of pulmonary abnormalities in COVID-19 patients on chest X-rays (CXRs).

Methodology: Cross-sectional study used retrospective data from August 2021 to February 2022, 119 chest radiographs from PCR confirmed COVID-19 patients in Chaiyaphum hospital were reviewed by 3 radiologists. Three radiologists, with 11 and 9 years of experience, as well as one new radiologist, independently reviewed all chest X-rays using the Brixia scoring system, which divides the CXR into six lung zones. One radiologist reviewed all CXRs twice to evaluate intra-observer reliability. The inter-observer agreement was use weighted Cohen’s Kappa, the Intra-class correlation coefficients in the overall and zone-specific were calculated and along with 95% confidence interval (95%CI).

Results: This study found the overall inter-observer agreement between radiologist show substantial to almost perfect (weighted Cohen’s κ = 0.79-0.90).  Concordance in interpretation between experienced radiologists was slightly better than concordance in interpretation between experienced and new radiologists. Intra-class correlation coefficients (ICC) in the overall show good correlation (ICC 0.88:95%CI=0.84-0.91). The intra-observer reliability of one experienced radiologist shows very good correlation (ICC 0.97: 95%CI=0.95-0.98 and Pearson correlation coefficient =0.97, p <0.001).

Conclusions: Substantial to almost perfect inter-observer agreement between experience and new radiologist in COVID-19 chest radiograph interpretation.

References

Bogoch II, Watts A, Thomas-Bachli A, Huber C, Kraemer MUG, Khan K. Pneumonia of unknown aetiology in Wuhan, China: potential for international spread via commercial air travel. J Travel Med 2020;27(2):1-3. doi: 10.1093/jtm/taaa008

World Health Organization. Pneumonia of unknown cause – China. [cited Dec 13, 2020]. Available from: https://www.who.int/csr/don/05-january-2020-pneumonia-of-unkown-causechina/en/.

Zu ZY, Jiang MD, Xu PP, Chen W, Ni QQ, Lu GM, et al. Coronavirus disease 2019 (COVID-19): a perspe+ctive from China. Radiology 2020;296(2):E15-E25. doi: 10.1148/radiol.2020200490

Ng MY, Lee EYP, Yang J, Yang F, Li X, Wang H, et al. Imaging profile of the COVID-19 infection: radiologic findings and literature review. Radiol Cardiothorac Imaging 2020;2(1): e200034. doi: 10.1148/ryct.2020200034

Bernheim A, Mei X, Huang M, Yang Y, Fayad ZA, Zhang N, et al. Chest CT findings in coronavirus disease-19 (COVID-19): relationship to duration of infection. Radiology 2020; 295(3):200463. doi: 10.1148/radiol.2020200463

Borghesi A, Maroldi R. COVID-19 outbreak in italy: experimental chest x-ray scoring system for quantifying and monitoring disease progression. Radiol Med 2020;125(5):509-13. doi: 10.1007/s11547-020-01200-3

World Health Organization. WHO global epidemiological surveillance standards for influenza. In. Edited by Influenza WHOWGESSf. Geneva, Switzerland: World Health Organization; 2013

Warren MA, Zhao Z, Koyama T, Bastarache JA, Shaver CM, Semler MW, et al. Severity scoring of lung oedema on the chest radiograph is associated with clinical outcomes in ARDS. Thorax 2018;73(9):840-6. doi: 10.1136/thoraxjnl-2017-211280

Monaco CG, Zaottini F, Schiaffino S, Villa A, Della Pepa G, Carbonaro LA, et al. Chest x-ray severity score in COVID-19 patients on emergency department admission: a two-centre study. Eur Radiol Exp 2020;4(68):1-7. doi: 10.1186/s41747-020-00195-w

Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med 1998;17(1):101-10. doi: 10.1002/(sici)1097-0258(19980115)17:1<101::aid-sim727>3.0.co;2-e.

Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33(1):159-74.

Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 2016;15(2):155-63. doi: 10.1016/j.jcm.2016.02.012

Van Assen M, Zandehshahvar M, Maleki H, Kiarashi Y, Arleo T, Stillman AE, et al. COVID-19 pneumonia chest radiographic severity score: variability assessment among experienced and in-training radiologists and creation of a multireader composite score database for artificial intelligence algorithm development. Br J Radiol 2022;95(1134):20211028. doi: 10.1259/bjr.20211028

Cohen JP, Dao L, Roth K, Morrison P, Bengio Y, Abbasi AF, et al. Predicting COVID-19 pneumonia severity on chest x-ray with deep learning. Cureus 2020;12(7): e9448. doi: 10.7759/cureus.9448

Moncada DC, Rueda ZV, Macías A, Suárez T, Ortega H, Vélez LA. Reading and interpretation of chest X-ray in adults with community-acquired pneumonia. Braz J Infect Dis 2011;15(6):540-6.

Published

2024-02-19

How to Cite

1.
Kunchai W. Inter-observer Reliability for Chest X-Ray Interpretation in COVID-19 Patients of Radiologists with Different Experiences. SRIMEDJ [Internet]. 2024 Feb. 19 [cited 2024 Dec. 4];39(1):74-82. Available from: https://li01.tci-thaijo.org/index.php/SRIMEDJ/article/view/260976

Issue

Section

Original Articles