处理缺失值的常规方法总结

  • A+

处理缺失值的常规方法总结

来源:数据人网

地址:http://www.shujuren.org/article/117.html

title: “处理缺失值的常规方法总结”

author: “Cynthia Li, CFA

date: “May 6, 2016”

前言

  • 现实生活中的数据是纷繁杂乱的,收集来的数据有缺失和录入错误司空见惯,所以学习如果处理这些常见问题是每一个数据人必须掌握的技能,俗话说巧妇难为无米之炊,不能很好的处理原始数据会给后来的建模带来麻烦,甚至引入不必要的偏差和错误,数据科学家都熟悉“垃圾进垃圾出”的说法,今天让我们来学习成为合格数据人的必修课 —- 处理缺失值的常规办法。

数据介绍

  • 我们继续使用上一节的贷款数据集(数据内容请见下篇介绍
  1. library(foreign)
  2. library(gmodels)
  3. library(mice)
  4. library(VIM)
  5. library(Hmisc)
  6. library(DMwR)
  7. loan.data = read.spss("Loan_ROC.sav", to.data.frame=TRUE)
  8. levels(loan.data$education)
  9. loan.data$education=factor(loan.data$education)
  10. levels(loan.data$education)

引入缺失值

  1. set.seed(1)
  2. data.copy = loan.data
  3. data.copy$debt_income[sample(1:nrow(data.copy), 10)] <- NA
  4. data.copy$education[sample(1:nrow(data.copy), 10)] <- NA
  5. levels(data.copy$education)
  6. data.copy$education=factor(data.copy$education)
  7. levels(data.copy$education)

MICE 包 —— 检查缺失值分布

  • 注意连续型数据和分类型数据的区别
  • 两类缺失值
    • MCAR: missing completely at random (desirable scenario; safe maximum threshold — 5% of total large dataset)
    • MNAR: missing not at random (wise and worthwhile to check data gathering process)
  • 检查行与列(自建公式)
  1. summary(data.copy)
  2. pMiss <- function(x){sum(is.na(x))/length(x)*100}
  3. apply(data.copy, 2, pMiss)
  4. apply(data.copy, 1, pMiss)
  5. md.pattern(data.copy)
  • r names(md.pattern(data.copy)[,1][1]) 个完整值
  • r names(md.pattern(data.copy)[,1][2]) 个观察值只缺失r names(md.pattern(data.copy)[1,][12])
  • r names(md.pattern(data.copy)[,1][3]) 个观察值只缺失r names(md.pattern(data.copy)[1,][13])
  • r names(md.pattern(data.copy)[,1][4]) 个观察值缺失前两个变量

VIM 包 + box plot 箱线图 —- 缺失值视觉化

  1. aggr_plot <- aggr(data.copy, col=c('blue','red'), numbers=TRUE, sortVars=TRUE, labels=names(data.copy), cex.axis=.7, gap=3, ylab=c("Missing data histogram","Missing value pattern"))
  2. marginplot(data.copy[,c(5,2)], main='Missing value box plot\n Constrain: 2 variables')
  • debt_income 的红 - 缺失 education 的分布
  • debt_income 的蓝 - 其余不缺失 education 的分布
  • 希望看到两个变量分别的红和蓝条分布相似(注意数据类型的不同)

1. 处理方法 —- 删除行/列 (na.action=na.omit)

适用条件:

    1. 足够大数据集 (model doesn’t lose power)
    1. 不会引入偏差 (no disproportionate or non-representation of classes)
    1. 删列:不起重要预测作用的变量

2. 处理方法 —- 替换 (mean / median / mode)

适用条件:

    1. 不需要非常精确的估算
    1. 变量 variation is low,或者 low leverage over response variable
  1. impute(data.copy$debt_income, mean) # replace with mean, indicated by star suffix
  2. impute(data.copy$debt_income, 20) # replace with specific number
  3. # data.copy$debt_income[is.na(data.copy$debt_income)] <- mean(data.copy$debt_income, na.rm = T) # impute manually

计算正确率 —- 替换 mean

  1. actuals <- loan.data$debt_income[is.na(data.copy$debt_income)]
  2. predicteds <- rep(mean(data.copy$debt_income, na.rm=T), length(actuals))
  3. regr.eval(actuals, predicteds)

3. 处理方法 —- 推测 (高大上的方法:kNN, rpart, and mice)

3.1. kNN

  1. # 推测方法:identify ‘k’ closest observations based on euclidean distance and computes the weighted average (weighted based on distance) of these ‘k’ obs.
  2. # 优点:can impute all missing values in all variables with one call to function. It takes whole dataframe as argument and don’t have to specify which variable to impute.
  3. # 注意:not to include response variable.
  1. knnOutput <- knnImputation(data.copy)
  2. anyNA(knnOutput)
  3. actuals <- loan.data$debt_income[is.na(data.copy$debt_income)]
  4. predicteds <- knnOutput[is.na(data.copy$debt_income), "debt_income"]
  5. regr.eval(actuals, predicteds)
  • The mean absolute percentage error (mape) has improved by ~ 65% compared to the imputation by mean.

3.2. rpart

  1. # 优点:比较于 kNN,rpart 和 mice 可以用于分类型数据;而且 rpart 仅需要一个 predictor variable to be non-NA
  2. # 注意:not to include response variable.
  3. # `method=class` for factor variable
  4. # `method=anova` for numeric variable
  1. library(rpart)
  2. class_mod <- rpart(education ~ ., data=data.copy[!is.na(data.copy$education), ], method="class", na.action=na.omit)
  3. anova_mod <- rpart(debt_income ~ ., data=data.copy[!is.na(data.copy$debt_income), ], method="anova", na.action=na.omit)
  4. education_pred <- predict(class_mod, data.copy[is.na(data.copy$education), ],type='class')
  5. debt_income_pred <- predict(anova_mod, data.copy[is.na(data.copy$debt_income), ])
  6. actuals <- loan.data$debt_income[is.na(data.copy$debt_income)] # debt_income accuracy
  7. predicteds <- debt_income_pred
  8. regr.eval(actuals, predicteds)
  9. actuals <- loan.data$education[is.na(data.copy$education)]
  10. mean(actuals != education_pred) # misclass error for education accuracy
  • debt_income’s accuracy is slightly worse than kNN but still much better than imputation by mean

3.3. mice

  1. # 方法:2-step:mice() to build multiple models; complete() to generate one/several completed data (default is first).
  1. miceMod <- mice(data.copy[, !names(data.copy) %in% "Loan"], method="rf") # based on random forests
  2. miceOutput <- complete(miceMod) # generate completed data.
  3. anyNA(miceOutput)
  4. actuals <- loan.data$debt_income[is.na(data.copy$debt_income)]
  5. predicteds <- miceOutput[is.na(data.copy$debt_income), "debt_income"]
  6. regr.eval(actuals, predicteds) # debt_income accuracy
  7. actuals <- loan.data$education[is.na(data.copy$education)]
  8. predicteds <- miceOutput[is.na(data.copy$education), "education"]
  9. mean(actuals != predicteds) # misclass error education accuracy
  • The mean absolute percentage error (mape) has improved by ~ 62% compared to the imputation by mean.
  • Mis-classification error is the same as rpart’s 60%, which may be contributed to imbalanced classes and small sample.
  1. miceMod <- mice(data.copy[, !names(data.copy) %in% "Loan"], method="pmm") # based on predictive mean matching
  2. miceOutput <- complete(miceMod) # generate completed data.
  3. anyNA(miceOutput)
  4. actuals <- loan.data$debt_income[is.na(data.copy$debt_income)]
  5. predicteds <- miceOutput[is.na(data.copy$debt_income), "debt_income"]
  6. regr.eval(actuals, predicteds) # debt_income accuracy
  7. actuals <- loan.data$education[is.na(data.copy$education)]
  8. predicteds <- miceOutput[is.na(data.copy$education), "education"]
  9. mean(actuals != predicteds) # misclass error education accuracy
  • The mean absolute percentage error (mape) has improved by ~ 61% compared to the imputation by mean.
  • But mis-classification error is 50%, better than previous two methods.

4. 数据视觉化检测 —- 对比原始数据和推测值

4.1. scatterplot

  1. xyplot(miceMod,debt_income ~ age + year_emp + income, pch=18,cex=0.8)
  • Desirable scenario: magenta points (imputed) matches shape of blue (observed). Matching shape 说明 imputed values are indeed “plausible values”.

4.2. density plot

  1. densityplot(miceMod)
  • Magenta - density of imputed data for each imputed dataset
  • Blue - density of observed data
  • Desirable: similar distributions

4.3. stripplot plot

  1. stripplot(miceMod, pch = 20, cex = 1.2)
  • 点状图

5. 推测处理后的数据建模 —- pooling

  1. modelFit1 <- with(miceMod,lm(debt_income ~ age + year_emp + income))
  2. summary(pool(modelFit1))
  • fmi —- fraction of missing information
  • lambda —- proportion of total variance that is attributable to missing data

6. 备注(额外介绍)—- 处理 random seed initialization

  • mice function is initialed with a specific seed, so results are dependent on initial choice. To reduce this effect, we impute a higher number of dataset, by changing default m=5 parameter in mice() function
    1. tempData2 <- mice(data.copy[, !names(data.copy) %in% "Loan"], method='pmm',m=50, maxit=10, seed=0)
    1. modelFit2 <- with(tempData2,lm(debt_income ~ age + year_emp + income))
    2. summary(pool(modelFit2))
南霁月
MySQL必知必会
基于大数据的用户特征分析
误差分位数的默示有效估计与\ 自回归时间序列的预测区间
中国大数据生态图谱&大数据交易市场专题研究报告

发表评论

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: