Comparison of EM algorithm and standard imputation methods for missing data: A questionnaire study on diabetic patients

Abstract

Background and Objectives: Missing data is a big challenge in the research. According to the type of the study and of the variables, different ways have been proposed to work with these data. This study compared five popular imputation approaches in addressing missing data in the questionnaires. Methods: In this study, 500 questionnaires were used for self-medication in diabetic patients. Missing in the observations was artificially generated by random selection of questions and then deleting them. Five imputation ways included: 1) the mean of the questions, 2) the mean of the person, 3) the mode of the person, 4) linear regression, and 5) EM algorithm. For each method, the mean and standard deviation were compared with imputation. The Spearman correlation coefficient, the percentage of incorrectly classified and kappa statistic were also calculated. Results: A kappa higher than 0.81 represented almost perfect agreement at 10 missingness. The EM algorithm showed the highest level of agreement with the results of actual data with a Kappa of 0.886. With increasing missingness to 30, the EM algorithm and the mean of the person showed a rather similar agreement with a Kappa of 0.697 and 0.687, respectively. Conclusion: In this study, the EM algorithm was the most accurate method for handling missing data in all models. The mean of the person method is easy for handling missing data, especially for most non statisticians. © 2015, Iranian Epidemiological Association. All rights reserved.