An extra column with constant number leads to different imputed result

[zhxiaokang]

I accidentally added an extra (useless) column with sample ID to my dataset, then "mice" gave me a totally different imputed result. I also tested adding one column with a constant number which had the same issue. Why does this kind of extra column affect the imputation result?

Here is a test code:

library(mice)
library(naniar)  # to replace some values with NA
library(compare)  # to compare two variables

set.seed(123)
df.origin <- data.frame(replicate(10, sample(0:100, 1000, rep=TRUE)))

# replace the 0 with NA
df.origin.na <- df.origin %>% replace_with_na_all(condition = ~.x == 0)

# add an extra column of sample ID
df.with.id.na <- data.frame(id = seq(1, 1000), df.origin.na)

# add an extra column of constant number
df.with.constant.na <- data.frame(constant = rep(1, 1000), df.origin.na)

# Do the imputation
df.origin.imputed <- complete(mice(df.origin.na, m=1, maxit = 10, method = 'pmm', seed = 500))
df.with.id.imputed <- complete(mice(df.with.id.na, m=1, maxit = 10, method = 'pmm', seed = 500))
df.with.constant.imputed <- complete(mice(df.with.constant.na, m=1, maxit = 10, method = 'pmm', seed = 500))

compare(df.origin.imputed, df.with.id.imputed)
# output: FALSE, indicating that these two variables are different

compare(df.origin.imputed, df.with.constant.imputed)
# output: FALSE, indicating that these two variables are different

[stefvanbuuren]

I wouldn't expect the results to be identical.

• In case 1, you have added id to the imputation model for every variable, so the results will be different. Also, in compare, the first argument has 10 columns and the second one has 11. The following code rectifies both problems.

pred <- make.predictorMatrix(df.with.id.imputed)
pred[, 1] <- 0
df.with.id.imputed2 <- complete(mice(df.with.id.na, predictorMatrix = pred, m=1, maxit = 10, method = 'pmm', seed = 500))
compare(df.origin.imputed, df.with.id.imputed2[, -1])
# TRUE

• Likewise the following code addresses case 2:

compare(df.origin.imputed, df.with.constant.imputed[, -1])
# TRUE

I assume this answers your questions. Thanks for your interest in mice.

[zhxiaokang]

Thank you! I'm so sorry that the compare command was a mistake. As you said, the extra column should have been removed to do the compare.

Now as I understand, adding a column of constant won't change the result of imputation, but adding a column of "ID" will change the imputation, so what you did with make.predictorMatrix is to tell mice that when doing imputation, the column id shouldn't be taken into consideration, right?

But I'm still wondering: when doing imputation, shouldn't the columns (variables/attributes/features) be independent of each other and the missing values should be inferred by the values from other rows of the same column (supposing that each row is a sample and each column is a feature)?

[stefvanbuuren]

In order to create high-quality imputations we need to account for both the values within the same row and the values within the same column of each missing cell. If we would focus on column values only (e.g. by imputing the mean or by taking a random sample from the observed values from that column) the correlations between the variables after imputation will systematically go down. The role of the imputation model is to preserve the relations in the data as well as the uncertainty about these relations. See the discussion in sections 1.3.3-1.3.5 in https://stefvanbuuren.name/fimd/sec-simplesolutions.html#sec:meanimp for background.

[zhxiaokang]

Thank you Stef! That perfectly answers my question.