3

I am working with many GDAL rasters in memory in Python. Users on forums suggest using either one of two formats; 'VRT' or 'MEM'. I am confused about the difference between GDAL's 'MEM' format and GDAL's 'VRT' format however. I want to process many images with gdal.Warp, and make sure that I clear memory between each image to prevent leakage. Let's say I use warp and want to write the results to a location in memory as such:

vrtresult = gdal.Warp('/vsimem/swir.vrt', swir, options=upsample_options, format="VRT")

When I subsequently want to remove the swir.vrt from memory, I am supposed to use gdal.Unlink('/vsimem/swir.vrt'). Afterwards however vrtresult still contains the gdal object, which suggests to me that it is more than a handle referring to a location in-memory. If that is the case, does it store the result in memory twice? If I use the memory format however, for example as such;

memresult = gdal.Warp('/vsimem/swir.tif', swir, options=upsample_options, format="MEM")

I cannot even unlink. Using gdal.Unlink('/vsimem/swir.tif') will return "RuntimeError: unknown error occurred" (when you have gdal.UseExceptions() enabled). The 'MEM' format seems to be made just for the application I mention above, though there is no clear way of clearing memory. So I would like to know what the difference is between the two, and which would be cleanest to use if I want to temporarily do something in-memory.

TheBackwardsman
  • 33
  • 1
  • 11
  • If you mean this vrt https://gdal.org/drivers/raster/vrt.html it is like a recipe "take input file, apply coordinate transformation" and as a result you can open the vrt file and get a re-projected result without making a physical copy of the data. About mem and vsimem, perhaps this helps https://gis.stackexchange.com/questions/397451/why-does-vsimem-have-a-path. Python gotchas are also worth reading https://gdal.org/api/python_gotchas.html. – user30184 Feb 17 '23 at 15:58

1 Answers1

3

Ok, one by one.

There is nothing preventing you (well, there is locking, but by default it is not enabled) from deleting the file on disk (or in memory) after opening a Dataset. What will be the result? It is undefined in the general case, so don't do that. In some cases it might work as the file is still on the disk while there are open file handles, in some other cases, it might lead to a crash.

The Python implementation of GDAL does not support explicitly closing datasets. Normally the Python GC does this once you are finished with it. What you can do it to simply set the reference to None before unlinking:

memresult = None

See also Why close a dataset in GDAL Python?

Globally, this is a problem that remains somewhat unsolved in Python. I am the author of the Node.js implementation that is very similar and I have a special form of closing the Dataset when it is in memory to avoid this kind of problems.

Do not worry about leaks, if you are able to unlink the dataset from /vsimem/ and close the in-memory reference, everything should be ok.

On the difference between the formats:

  • .VRT is a meta-format that can mosaic a single dataset view from multiple datasets - it can be used both in memory and on the disk
  • .MEM is a in-memory format that stores the information raw, it is probably the fastest-one but does not support compression, does not have a filename and does not need to be deleted, closing the dataset frees the memory
  • /vsimem/ allows you to store in memory any format - it is a form of RAM disk - and it is a very versatile choice - interacting with these files uses normal file I/O but redirected to memory

.MEM cannot be stored on disk, but .VRT can be stored in /vsimem/.

Criminal_Affair_At_SO
  • 311
  • 1
  • 3