| Name: | Description: | Size: | Format: | |
|---|---|---|---|---|
| 1.96 MB | Adobe PDF |
Authors
Abstract(s)
Atualmente lidamos com um grande volume de dados e vários programas que permitem
fazer análise destes dados. No entanto, os valores omissos representam um problema
frequente no processo de análise destes conjuntos de dados que podem surgir por vários
motivos. Por exemplo, podem ser resultados perdidos das análises duma amostra, ou
alguns indivíduos não responderem a um determinado questionário. Visto que a maior
parte dos programas e algoritmos utilizados para o tratamento de dados requere conjuntos de dados sem valores omissos, isto é, dados completos, a sua existência pode limitar a
análise dos dados. Daí, surge a necessidade de recorrer a métodos de imputação de valores omissos. Nesta dissertação foram utilizados e comparados seis métodos distintos de
imputação, disponíveis no software R e avaliado o seu desempenho em conjuntos de dados relacionados com a área da educação, nomeadamente dados da avaliação nacional do
rendimento escolar (Prova Brasil). Foi estudada uma amostra de 20408 estudantes para
testar os seis algoritmos em quatro subconjuntos de dados gerados por simulação com
diferentes percentagens de valores omissos, considerando 5%, 10%, 15% e 20% nas variáveis de interesse. Foram explorados métodos de imputação simples (Média, Mediana e
Moda), métodos baseados em aprendizagem automática (kNN e bPCA) e um método de
imputação múltipla (MICE). Foi avaliado o desempenho de cada método adotado neste
trabalho calculando os respetivos erros de imputação através as métricas RMSE e MAE.
Os resultados obtidos mostram que o método de imputação pela Moda forneceu quase de
forma constante menores valores de erro.
We currently deal with a large volume of data and several programs that allow analysis of this data. However, missing values represent a frequent problem in the process of analyzing these data sets, which can arise for several reasons. For example, they may be missing results from a sample analysis, or some individuals may not answer a questionnaire. Since most programs and algorithms used for data processing require data sets without missing values, that is, complete data, their existence can limit data analysis. Hence, the need arises to use methods for imputing missing values. In this dissertation, six different imputation methods, available in software R, were used compared. Their performance was evaluated in datasets related to the education area, namely data from the national evaluation of school performance (Prova Brasil). A sample of 20408 students was studied to test the six algorithms in four subsets of data with different percentages of missing values, considering 5%, 10%, 15% and 20% in the variables of interest. Single imputation methods (Mean, Median and Mode), methods based on machine learning (kNN and bPCA) and a multiple imputation method (MICE) were explored. The performance of each method adopted in this work was evaluated by calculating the respective imputation errors using the metrics RMSE and MAE. The results obtained show that the method of imputation by Mode provided almost constantly lower values of error.
We currently deal with a large volume of data and several programs that allow analysis of this data. However, missing values represent a frequent problem in the process of analyzing these data sets, which can arise for several reasons. For example, they may be missing results from a sample analysis, or some individuals may not answer a questionnaire. Since most programs and algorithms used for data processing require data sets without missing values, that is, complete data, their existence can limit data analysis. Hence, the need arises to use methods for imputing missing values. In this dissertation, six different imputation methods, available in software R, were used compared. Their performance was evaluated in datasets related to the education area, namely data from the national evaluation of school performance (Prova Brasil). A sample of 20408 students was studied to test the six algorithms in four subsets of data with different percentages of missing values, considering 5%, 10%, 15% and 20% in the variables of interest. Single imputation methods (Mean, Median and Mode), methods based on machine learning (kNN and bPCA) and a multiple imputation method (MICE) were explored. The performance of each method adopted in this work was evaluated by calculating the respective imputation errors using the metrics RMSE and MAE. The results obtained show that the method of imputation by Mode provided almost constantly lower values of error.
Description
Keywords
Análise Descritiva de Dados Bpca Imputação Knn Média Médiana Mice Moda Valores Omissos