OPTIMAL CUT-OFF POINTS FOR RECEIVER OPERATING CHARACTERISTIC (ROC) CURVE ANALYSIS IN DEVELOPING TOOLS OF HEALTH INNOVATIONS: EXAMPLE USING STATA

Authors

  • Pongdech Sarakarn Epidemiology and Biostatistics Department, Faculty of Public Health, Khon Kaen University, Thailand
  • Pattaranan Munpolsri ASEAN Caner Epidemiology and Prevention Research Group (ACEP), Faculty of Public Health, Khon Kaen University, Khon Kaen

DOI:

https://doi.org/10.14456/tbps.2021.7

Keywords:

receiver operating characteristics (ROC), optimal cut-off point, Youden index, area under the curve (AUC), health innovation

Abstract

Receiver Operating Characteristic (ROC) curve analysis is a method that is used quite widely in Diagnosis of disease, but for use in the development of health innovations it is rarely seen, and its possible application in this area remains unclear. This article presents the principles and indicators that are used for assessing the accuracy of diagnostic tests, and considers three optimal cut-off point methods, including - Euclidian’s index, Youden’s index and Weighted Number Needed to Misdiagnose (Weighted NNM). In addition, this work assesses the overall discriminant accuracy of the tests, for their potential application in health innovation - assessing each method individually and comparatively, using the area below the ROC curve (AUC). Examples were demonstrated using commands in STATA software programme. The results and related statistics were presented and interpreted as well. Finally, the implications of using ROC analysis were identified, which cover both the advantages and limitations of the ROC analysis.

References

Habibzadeh F, Habibzadeh P, Yadollahie M. On determining the most appropriate test cut-off value: the case of tests with continuous results. Biochem Med (Zagreb). 2016;26(3):297-307.

Tripepi G, Jager KJ, Dekker FW, Zoccali C. Diagnostic methods 2: receiver operating characteristic (ROC) curves. Kidney Int. 2009;76(3):252-6.

Ren Q, Su C, Wang H, Wang Z, Du W, Zhang B. Prospective study of optimal obesity index cut-off values for predicting incidence of hypertension in 18-65-year-old Chinese adults. PloS one. 2016;11(3):e0148140-e0148140.

Silva PAB, Soares SM, Santos JFG, Silva LB. Cut-off point for WHOQOL-bref as a measure of quality of life of older adults. Rev Saude Publica. 2014;48(3):390-7.

Westergren A, Norberg E, Vallén C, Hagell P. Cut-off scores for the Minimal Eating Observation and Nutrition Form – Version II (MEONF-II) among hospital inpatients. Food Nutr Res. 2011;55(1):7289.

Sung YH, Cho MS, Kwon IG, Jung YY, Song MR, Kim K, et al. Evaluation of falls by inpatients in an acute care hospital in Korea using the Morse Fall Scale. Int J Nurs Pract. 2014;20(5):510-7.

Kim HR, Kim HS. Optimal cutoffs of cardiometabolic risk for postmenopausal Korean women. Asian Nurs Res (Korean Soc Nurs Sci). 2017;11(2):107-12.

Chan KTM, Cheung AHN. Application of receiver operating characteristic (ROC) curve to determine the diagnostic ability of a validated ten-item questionnaire (SS - 10) in estimating the prevalence of sensitive skin in Hong Kong population. Int J Inn Res Med Sci. 2019;4(07):405-13.

Sidey-Gibbons JAM, Sidey-Gibbons CJ. Machine learning in medicine: a practical introduction. BMC Med Res Methodol. 2019;19(1):64.

Kumar R. Indrayan A. Receiver operating characteristic (ROC) curve for medical researchers. Indian Pediatr. 2011;48(4):277-87.

Zou KH, O’Malley AJ, Mauri L. Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation. 2007;115(5):654-7.

Hajian-Tilaki K. The choice of methods in determining the optimal cut-off value for quantitative diagnostic test evaluation. Stat Methods Med Res. 2018;27(8):2374-83.

Perkins NJ, Schisterman EF. The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. Am J Epidemiol. 2006;163(7):670-5.

Habibzadeh F, Habibzadeh P, Yadollahie M. Criterion used for determination of test cut-off value. Diabetes Res Clin Pract. 2017;128:138-9.

Habibzadeh F, Yadollahie M. Number needed to misdiagnose: a measure of diagnostic test effectiveness. Epidemiology. 2013;24(1):170.

DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837-45.

Metz CE. Basic principles of ROC analysis. Semin Nucl Med. 1978;8(4):283-98.

Ray P, Manach YL, Riou B, Houle TT. Statistical Evaluation of a Biomarker. Anesthesiology. 2010;112(4):1023-40.

Mandrekar JN. Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol. 2010;5(9): 1315-16.

Kamarudin AN, Cox T, R Kolamunnage-Dona. Time-dependent ROC curve analysis in medical research: current methods and applications. BMC Med Res Methodol. 2017;17(1):53.

Pepe M, Longton G, Janes H. Estimation and comparison of receiver operating characteristic curves. Stata J. 2009;9(1):1.

Demler OV, Pencina MJ, D'Agostino RB Sr. Misuse of DeLong test to compare AUCs for nested models. Stat Med. 2012;31(23):2577-87.

Downloads

Published

2021-05-18

Issue

Vol. 16 No. 1 (2021): JANUARY - JUNE 2021

Section

Review Articles

License

All articles published and information contained in this journal such as text, graphics, logos and images is copyrighted by and proprietary to the Thai Bulletin of Pharmaceutical Sciences, and may not be reproduced in whole or in part by persons, organizations, or corporations other than the Thai Bulletin of Pharmaceutical Sciences and the authors without prior written permission.