Inter-observer Reliability for Chest X-Ray Interpretation in COVID-19 Patients of Radiologists with Different Experiences
Keywords:
Interobserver reliability, COVID-19 CXR, COVID pneumonia severity scoreAbstract
Background and Objectives: Radiographs play a very important role in diagnosing and planning patient care. This study aimed to assess the inter- and intraobserver reliability among radiologists, particularly between new and experienced radiologists, in evaluating the extent of pulmonary abnormalities in COVID-19 patients on chest X-rays (CXRs).
Methodology: Cross-sectional study used retrospective data from August 2021 to February 2022, 119 chest radiographs from PCR confirmed COVID-19 patients in Chaiyaphum hospital were reviewed by 3 radiologists. Three radiologists, with 11 and 9 years of experience, as well as one new radiologist, independently reviewed all chest X-rays using the Brixia scoring system, which divides the CXR into six lung zones. One radiologist reviewed all CXRs twice to evaluate intra-observer reliability. The inter-observer agreement was use weighted Cohen’s Kappa, the Intra-class correlation coefficients in the overall and zone-specific were calculated and along with 95% confidence interval (95%CI).
Results: This study found the overall inter-observer agreement between radiologist show substantial to almost perfect (weighted Cohen’s κ = 0.79-0.90). Concordance in interpretation between experienced radiologists was slightly better than concordance in interpretation between experienced and new radiologists. Intra-class correlation coefficients (ICC) in the overall show good correlation (ICC 0.88:95%CI=0.84-0.91). The intra-observer reliability of one experienced radiologist shows very good correlation (ICC 0.97: 95%CI=0.95-0.98 and Pearson correlation coefficient =0.97, p <0.001).
Conclusions: Substantial to almost perfect inter-observer agreement between experience and new radiologist in COVID-19 chest radiograph interpretation.
References
Bogoch II, Watts A, Thomas-Bachli A, Huber C, Kraemer MUG, Khan K. Pneumonia of unknown aetiology in Wuhan, China: potential for international spread via commercial air travel. J Travel Med 2020;27(2):1-3. doi: 10.1093/jtm/taaa008
World Health Organization. Pneumonia of unknown cause – China. [cited Dec 13, 2020]. Available from: https://www.who.int/csr/don/05-january-2020-pneumonia-of-unkown-causechina/en/.
Zu ZY, Jiang MD, Xu PP, Chen W, Ni QQ, Lu GM, et al. Coronavirus disease 2019 (COVID-19): a perspe+ctive from China. Radiology 2020;296(2):E15-E25. doi: 10.1148/radiol.2020200490
Ng MY, Lee EYP, Yang J, Yang F, Li X, Wang H, et al. Imaging profile of the COVID-19 infection: radiologic findings and literature review. Radiol Cardiothorac Imaging 2020;2(1): e200034. doi: 10.1148/ryct.2020200034
Bernheim A, Mei X, Huang M, Yang Y, Fayad ZA, Zhang N, et al. Chest CT findings in coronavirus disease-19 (COVID-19): relationship to duration of infection. Radiology 2020; 295(3):200463. doi: 10.1148/radiol.2020200463
Borghesi A, Maroldi R. COVID-19 outbreak in italy: experimental chest x-ray scoring system for quantifying and monitoring disease progression. Radiol Med 2020;125(5):509-13. doi: 10.1007/s11547-020-01200-3
World Health Organization. WHO global epidemiological surveillance standards for influenza. In. Edited by Influenza WHOWGESSf. Geneva, Switzerland: World Health Organization; 2013
Warren MA, Zhao Z, Koyama T, Bastarache JA, Shaver CM, Semler MW, et al. Severity scoring of lung oedema on the chest radiograph is associated with clinical outcomes in ARDS. Thorax 2018;73(9):840-6. doi: 10.1136/thoraxjnl-2017-211280
Monaco CG, Zaottini F, Schiaffino S, Villa A, Della Pepa G, Carbonaro LA, et al. Chest x-ray severity score in COVID-19 patients on emergency department admission: a two-centre study. Eur Radiol Exp 2020;4(68):1-7. doi: 10.1186/s41747-020-00195-w
Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med 1998;17(1):101-10. doi: 10.1002/(sici)1097-0258(19980115)17:1<101::aid-sim727>3.0.co;2-e.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33(1):159-74.
Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 2016;15(2):155-63. doi: 10.1016/j.jcm.2016.02.012
Van Assen M, Zandehshahvar M, Maleki H, Kiarashi Y, Arleo T, Stillman AE, et al. COVID-19 pneumonia chest radiographic severity score: variability assessment among experienced and in-training radiologists and creation of a multireader composite score database for artificial intelligence algorithm development. Br J Radiol 2022;95(1134):20211028. doi: 10.1259/bjr.20211028
Cohen JP, Dao L, Roth K, Morrison P, Bengio Y, Abbasi AF, et al. Predicting COVID-19 pneumonia severity on chest x-ray with deep learning. Cureus 2020;12(7): e9448. doi: 10.7759/cureus.9448
Moncada DC, Rueda ZV, Macías A, Suárez T, Ortega H, Vélez LA. Reading and interpretation of chest X-ray in adults with community-acquired pneumonia. Braz J Infect Dis 2011;15(6):540-6.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Srinagarind Medical Journal
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
