3

I have a few large LiDAR DEMs that I want to merge. Each DEM has some glitches around the borders which need to be removed in order to get a seamless final product. I manually traced around each DEM to create a polygon which excludes the glitchy areas. I am using gdalwarp to clip the raster files, but I am experiencing very slow performance.

The rasters are between 3 and 10 GB, which I understand is pretty big… however I have ~3 ghz of processing power & 4GB of RAM. These rasters load into QGIS quickly and are quick to refresh. I can derive a hillshade or slope raster from one of these DEMS in 2 or 3 minutes.. so it seems unreasonable when it takes 4-6 HOURS to clip one of these rasters with gdalwarp.

Can anybody offer an explanation of what might be going on?

I am also bothered by the very large file sizes (up to 4x larger than the original) which result from the clipping. Research suggests that this is a compression issue, but gdalinfo does not return any compression info for any of the original rasters, and even when I compress the files are still 2x larger than the originals. Could there be any reason for this other than compression?

corvus
  • 526
  • 7
  • 18
  • A test image and exact GDAL commands would help a lot. – user30184 Apr 16 '16 at 17:06
  • Things to consider are the filetype and compression of your original files, your chosen compression for your output raster and options for the hillshade stuff you did. That the files are refreshing quickly is probably due to pyramids. One general error is when your clipping polygon is in some parts larger than your original raster it takes really long. Also the functions might work probably a different way. Clip, as far a i understand it, puts a lot into RAM. And your files are larger then your (pretty small) Memory. And also after clipping the raster has to be interpolated due to pixel-shift. – Matte Apr 16 '16 at 17:08
  • The command I have been using is: gdalwarp -dstnodata 0 -q -cutline "clip.shp" -crop_to_cutline -dstalpha -tr 5.0 5.0 -of GTiff "county_dem.tif" "county_dem_clip.tif"

    DEMs can be found here, I am using Vernon, Crawford, Juneau, Sauk, Lacrosse, Monroe and Richland counties.

     – corvus Apr 16 '16 at 17:10
  • @Matte: Thanks for the response. I am sure that the functions which derive a hillshade or slope are much different than the gdalwarp function.. I was not sure if this could be a normal penomenon, but I thought it seemed unreasonable because of the huge difference in the time it takes to process an equivalent amount of data.

    "One general error is when your clipping polygon is in some parts larger than your original raster it takes really long"

    This could be the reason. I'll check back

     – corvus Apr 16 '16 at 17:12
  • the conversion from shapefile to raster is likely to take a lot of time in this process. – radouxju Apr 16 '16 at 17:13
  • 4Gb is barely enough to run an OS. An image-processing machine should have 8Gb or 16Gb RAM. – Vince Apr 16 '16 at 18:36
  • 2
    Shapefile is not converted into raster, it is used as a mask and that is not so expensive as far as I know. – user30184 Apr 16 '16 at 19:02
  • For performance gdalwarp to VRT first then gdal_translate with compression after that. See @Luke's answer to File size inflation normal with gdalwarp? – matt wilkie Nov 15 '18 at 17:43

0 Answers0