R - Dealing with missing data in SpatialPolygonDataFrames for Moran.test

Question

I'm trying to use Moran.test on a SpatialPolygonDataFrame consisting of 7194 elements in R. I know that there is around 150 polygons with NA values.

First I generate a spatial weights matrix:

W_cont_msoa <- poly2nb(all.data, queen=T) W_cont_msoa_mat <- nb2listw(W_cont_msoa, style="W", zero.policy=TRUE)

Then when I carry out Moran.test the code below returns:

> moran.test(residuals, listw=W_cont_msoa_mat, zero.policy=T)
Error in moran.test(residuals, listw = W_cont_msoa_mat, zero.policy = T) : 
objects of different length

I've also tried to this but it hasn't worked:

> moran.test(residuals, listw=W_cont_msoa_mat, zero.policy=T, na.rm=TRUE)
Error in moran.test(residuals, listw = W_cont_msoa_mat, zero.policy = T,  : 
unused argument(s) (na.rm = TRUE)

I've tried to go back to the original data and remove missing data using complete cases but I received the following:

> all.data[complete.cases(all.data)]
Error in all.data[complete.cases(all.data)] : 
error in evaluating the argument 'i' in selecting a method for function '[': Error in  
complete.cases(all.data) : invalid 'type' (S4) of argument

The structure of the data is as follows:

`Formal class 'SpatialPolygonsDataFrame' [package "sp"] with 5 slots
..@ data       :'data.frame':   7194 obs. of  23 variables:
.. ..$ MSOA01CD   : Factor w/ 7194 levels "E02000001","E02000002",..: 1 2 3 4 5 6 7 8    9 10 ...   
.. ..$ MSOA01NM   : Factor w/ 7194 levels "Adur 001","Adur 002",..: 1468 125 126 127     128 129 130 131 132 133 ...    
 .. ..$ averagesp  : num [1:7194] 11.6 20.9 20.3 21.4 19.1 ...  
 .. ..$ nga        : int [1:7194] 41 112 177 97 120 NA 142 161 122 113 ...  
 .. ..$ mediansp   : num [1:7194] 11.5 13.1 12.6 13.5 10 ...    
 .. ..$ mbps2      : int [1:7194] 125 74 104 48 83 NA 64 67 76 25 ...   
 .. ..$ msoa.total : int [1:7194] 2372 114 191 97 126 NA 157 208 130 119 ...    
 .. ..$ p.nga      : num [1:7194] 1.73 98.25 92.67 100 95.24 ...    
 .. ..$ p.mbps2    : num [1:7194] 5.27 64.91 54.45 49.48 65.87 ...  
 .. ..$ residents  : Factor w/ 4057 levels "10,003","10,004",..: 2222 1748 27 1261       3216 NA 3380 403 3093 3256 ... 
  .. ..$ area       : Factor w/ 6908 levels "1,002.23","1,002.58",..: 3707 2869 2840      3304 839 NA 1899 1327 6811 748 ...    
 .. ..$ density    : num [1:7194] 25.5 31.3 46.9 24.8 72.1 NA 50.6 81.5 87.4 76.8 ...   
  .. ..$ micro.bc   : num [1:7194] 36 32 83 33 39 NA 74 60 44 32 ...    
  .. ..$ small.bc   : num [1:7194] 49 14 64 3 14 NA 50 14 14 73 ... 
  .. ..$ medium.bc  : num [1:7194] 850 0 15 0 0 NA 10 0 0 0 ... 
  .. ..$ large.bc   : num [1:7194] 245 0 0 0 0 NA 0 0 0 0 ...   
   .. ..$ primary    : num [1:7194] 60 0 0 0 0 NA 0 0 0 0 ...   
  .. ..$ secondary  : num [1:7194] 135 0 20 0 0 NA 0 0 5 5 ...  
  .. ..$ tertiary   : num [1:7194] 16600 95 335 100 145 NA 300 200 160 80 ...   
  .. ..$ kibs       : num [1:7194] 11805 20 80 20 30 ...    
  .. ..$ edu        : num [1:7194] 2205 2681 811 2658 116 ...   
  .. ..$ p.non.white: num [1:7194] 21.4 35 45.4 19 33.7 NA 32.8 30.4 38.8 30.8 ...  
 .. ..$ claimants  : num [1:7194] 117 245 316 254 295 ...   
 ..@ polygons   :List of 7194   

Can anyone suggest how I overcome this problem please?

Thanks,

Ed

Answer 1

It is looking like the deleted rows in the @data slot are not propagating through the object. That is to say that the polygons associated with the deleted rows are not getting deleted. When you build a spatial weights matrix it is utilizing all the polygon objects thus, creating a mismatch in the dimension of the matrix and the number of actual observations in you regression. I believe that this is why the Moran's test is failing.

Here is a function that makes sure that removal of NA's propagates through all of the slots. The example is on the meuse point data but it will work on polygon (SpatialPolygonsDataFrame) objects as well.

# USE meuse AS EXAMPLE
require(sp)
  data(meuse)
  coordinates(meuse) <- ~x+y

# DISPLAY NA ROWS IN meuse  
meuse@data[!complete.cases(meuse@data),] 

# FUNCTION TO REMOVE NA's IN sp DataFrame OBJECT
#   x           sp spatial DataFrame object
#   margin      Remove rows (1) or columns (2) 
sp.na.omit <- function(x, margin=1) {
  if (!inherits(x, "SpatialPointsDataFrame") & !inherits(x, "SpatialPolygonsDataFrame")) 
    stop("MUST BE sp SpatialPointsDataFrame OR SpatialPolygonsDataFrame CLASS OBJECT") 
  na.index <- unique(as.data.frame(which(is.na(x@data),arr.ind=TRUE))[,margin])
    if(margin == 1) {  
      cat("DELETING ROWS: ", na.index, "\n") 
        return( x[-na.index,]  ) 
    }
    if(margin == 2) {  
      cat("DELETING COLUMNS: ", na.index, "\n") 
        return( x[,-na.index]  ) 
    }
 }

# DELETE NA's IN meuse AND SHOW CHANGE IN dim
meuse2 <- sp.na.omit(meuse)     
  dim(meuse)
    dim(meuse2) 

# PLOT DELETED POINTS IN RED    
plot(meuse, col="red", pch=20)
  plot(meuse2, col="black", pch=20, add=TRUE)