Adaptive Lasso sparse logistic regression on high-dimensional data with multicollinearity
Article Sidebar
Main Article Content
Abstract
A combination of high-dimensional sparse data and multicollinearity problems can lead to instabilities in a predictive model when applied to a new data set. The least absolute shrinkage and selection operator (Lasso) is widely employed in machine-learning algorithm for variable selection and parameter estimations. Although this method is computationally feasible for high-dimensional data, it has some drawbacks. Thus, the adaptive Lasso was developed using the adaptive weight on penalty function. This adaptive weight is related to the power order of the estimators. Hence, we focus on the power of adaptive weight on two penalty functions: adaptive Lasso and adaptive elastic net. This study aimed to compare the performances of the power of the adaptive Lasso and adaptive elastic net methods under high-dimensional sparse data with multicollinearity. Moreover, the performances of four penalized methods were compared: Lasso, elastic net, adaptive Lasso, and adaptive elastic net. They were compared using the mean of the predicted mean squared error for the simulation study and the classification accuracy for a real-data application. The results showed that the higher-order of the adaptive Lasso method performed best on very high-dimensional sparse data with multicollinearity when the initial weight was determined using a ridge estimator. However, in the case of high-dimensional sparse data with multicollinearity, the square root of the adaptive Lasso together with the initial weight using Lasso was the best option.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
References
Belsley, D. A., Kuh, E., & Welsch, R. E. (1980). Regression diagnostics: Identifying influential data and sources of collinearity. John Wiley & Sons.
Brimacombe, M. (2014). High-dimensional data and linear models: A review. Open Access Medical Statistics, 4, 17–27. https://doi.org/10.2147/OAMS.S56499
Cherkassky, V., & Mulier, F. (2007). Learning from data: Concepts, theory, and methods (2nd ed.). John Wiley & Sons.
Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004). Least angle regression. The Annals of Statistics, 32(2), 407–499. https://doi.org/10.1214/009053604000000067
Fan, J., & Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96(456), 1348–1360. https://doi.org/10.1198/016214501753382273
Golub, T. R., Slonim, D. K., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J. P., Coller, H., Loh, M. L., Downing, J. R., Caligiuri, M. A., Bloomfield, C. D., & Lander, E. S. (1999). Molecular classification of cancer: Class discovery and class prediction by gene expression monitoring. Science, 286(5439), 531–537. https://doi.org/10.1126/science.286.5439.531
Hardin, J., Garcia, S. R., & Golan, D. (2013). A method for generating realistic correlation matrices. The Annals of Applied Statistics, 7(3), 1733–1762. https://doi.org/10.1214/13-AOAS638
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer.
Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), 55–67.
Hosmer, D. W., Jr., Lemeshow, S., & Sturdivant, R. X. (2013). Applied logistic regression (3rd ed.). John Wiley & Sons.
Hosseinnataj, A., Bahrampour, A., Baneshi, M., Zolala, F., Nikbakht, R., Torabi, M., & Mazidi Sharaf Abadi, F. (2019). Penalized Lasso methods in health data: Application to trauma and influenza data of Kerman. Journal of Kerman University of Medical Sciences, 26(6), 440–449. https://doi.org/10.22062/jkmu.2019.89573
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning with applications in R. Springer.
Kastrin, A., & Peterlin, B. (2010). Rasch-based high-dimensionality data reduction and class prediction with applications to microarray gene expression data. Expert Systems with Applications, 37(7), 5178–5185. https://doi.org/10.1016/j.eswa.2009.12.074
Kleinbaum, D. G., & Klein, M. (2010). Logistic regression: A self-learning text (3rd ed.). Springer.
Le Cessie, S., & Van Houwelingen, J. C. (1992). Ridge estimators in logistic regression. Journal of the Royal Statistical Society: Series C (Applied Statistics), 41(1), 191–201. https://doi.org/10.2307/2347628
Makalic, E., & Schmidt, D. F. (2010). Review of modern logistic regression methods with application to small and medium sample size problems. In J. Li (Ed.), AI 2010: Advances in artificial intelligence (pp. 213–222). Springer. https://doi.org/10.1007/978-3-642-17432-2_22
Mukaka, M. M. (2012). Statistics corner: A guide to appropriate use of correlation coefficient in medical research. Malawi Medical Journal, 24(3), 69–71. https://pubmed.ncbi.nlm.nih.gov/23638278/
Pavlou, M., Ambler, G., Seaman, S., De Iorio, M., & Omar, R. Z. (2016). Review and evaluation of penalised regression methods for risk prediction in low-dimensional data with few events. Statistics in Medicine, 35(7), 1159–1177. https://doi.org/10.1002/sim.6782
Schaefer, R. L., Roi, L. D., & Wolfe, R. A. (1984). A ridge logistic estimator. Communications in Statistics - Theory and Methods, 13(1), 99–113. https://doi.org/10.1080/03610928408828664
Senaviratna, N. A. M. R., & Cooray, T. M. J. A. (2021). Multicollinearity in binary logistic regression model. Theory and Practice of Mathematics and Computer Science, 8, 11–19.
Sudjai, N., & Duangsaphon, M. (2020). Liu-type logistic regression coefficient estimation with multicollinearity using the bootstrapping method. Science, Engineering and Health Studies, 14(3), 203–214. https://doi.org/10.14456/sehs.2020.19
Sudjai, N., Siriwanarangsun, P., Lektrakul, N., Saiviroonporn, P., Maungsomboon, S., Phimolsarnti, R., Asavamongkolkul, A., & Chandhanayingyong, C. (2023a). Robustness of radiomic features: Two-dimensional versus three-dimensional MRI-based feature reproducibility in lipomatous soft-tissue tumors. Diagnostics, 13(2), Article 258. https://doi.org/10.3390/diagnostics13020258
Sudjai, N., Siriwanarangsun, P., Lektrakul, N., Saiviroonporn, P., Maungsomboon, S., Phimolsarnti, R., Asavamongkolkul, A., & Chandhanayingyong, C. (2023b). Tumor-to-bone distance and radiomic features on MRI distinguish intramuscular lipomas from well-differentiated liposarcomas. Journal of Orthopaedic Surgery and Research, 18(1), Article 255. https://doi.org/10.1186/s13018-023-03718-4
Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267–288.
Urgan, N. N., & Tez, M. (2008). Liu estimator in logistic regression when the data are collinear. In International Conference on Continuous Optimization and Knowledge-Based Technologies, Lithuania, Selected Papers (pp. 323–327). Vilnius.
Zou, H. (2006). The adaptive Lasso and its oracle properties. Journal of the American Statistical Association, 101(476), 1418–1429. https://doi.org/10.1198/016214506000000735
Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2), 301–320.
Zou, H., & Zhang, H. H. (2009). On the adaptive elastic-net with a diverging number of parameters. Annals of Statistics, 37(4), 1733–1751. https://doi.org/10.1214/08-AOS625
