2

I am currently working with GOES-R on AWS. I've downloaded recent data, which is in the netCDF file format. Specifically I am using band 3 (M3C03).

When loading it into GDAL, it is taking upwards of 5 minutes.

goes_dir = "goes"
goes_files = glob.glob(goes_dir + '/' + str(doy) + '/18/*M3C03*.nc')
goes_files[:3]

['goes/257/18/OR_ABI-L1b-RadF-M3C03_G16_s20172571815379_e20172571826146_c20172571826188.nc',
 'goes/257/18/OR_ABI-L1b-RadF-M3C03_G16_s20172571845379_e20172571856146_c20172571856196.nc',
 'goes/257/18/OR_ABI-L1b-RadF-M3C03_G16_s20172571830379_e20172571841146_c20172571841193.nc']

go = gdal.Open('NETCDF:' + goes_files[1] + ':Rad')
import time
start_time = time.time()
goes_arr = go.GetRasterBand(1).ReadAsArray()
print("--- %s seconds ---" % (time.time() - start_time))
# --- 277.96904397 seconds ---

What can I do to speed it up? The files are only 50 MB maximum.

Matthew Mage
  • 161
  • 9
  • 1
    You could look at using the netCDF library as shown here https://gis.stackexchange.com/questions/185352/transpose-large-netcdf-with-gdal-ncl-nco-or-cdo and also demonstrated to improve performance here http://lists.osgeo.org/pipermail/gdal-dev/2016-November/045573.html. That last link also suggests using GDAL translate to modify the raster so you could try that also – Liam G Nov 07 '17 at 00:33
  • 1
    Can you point some test data that can be downloaded directly? – user30184 Nov 07 '17 at 07:31
  • I linked to the AWS page. That's as direct as it gets with AWS. – Matthew Mage Nov 07 '17 at 18:39
  • @Freighty netCDF4-python opens the file effortlessly. GDALWarp still takes 5+ minutes to warp the netCDF, which is understandable considering its still using GDAL under the hood. – Matthew Mage Nov 07 '17 at 18:41
  • Can you just use the netCDF4 library to get the data into a numpy array and then convert to tif or other format? Such as here https://gis.stackexchange.com/questions/37238/writing-numpy-array-to-raster-file – Liam G Nov 08 '17 at 02:44
  • Are you still seeing the performance issue? If so, what version of GDAL are you using? I believe some enhancements were made with GDAL 2.2.x to better support GOES-R processing. However I have run into some performance issues with GDAL 2.3.x that I'm currently investigating. – greenlaw Apr 09 '19 at 22:10

1 Answers1

1

It doesn't look like there's an easy way to fix this. My solution was to warp it to a known projection GeoTIFF, which takes several minutes but only needs to be done once.

Matthew Mage
  • 161
  • 9