6

I have a simple script for processing thousands of rasters/shapefiles. i am not able to delete temporary files, which are created by processing. I can pass them to:

'memory:output_file'

but not for all of them, because sometimes I need to use gdal, which does not know memory. So I create gpkg/shapefile. But after 64 files (which are hold in memory, I get error from sqlite because it can not hold more than 64 open files...)

Sqlite ERROR: Too many connections: max 64

I am using standalone QgsApplication and calling processing.

The problem is, somehow the PyQGIS is locking files after processing. Similar problem here: QGIS: Release file lock on file used in a processing algorithm at the end of script

Another one here: Releasing PyQGIS file locks?

Another one: https://issues.qgis.org/issues/12502 (which says fixed/implemented)

And many others. Their solution does not work for me. If i try to put it into: 

QgsProject.instance().addMapLayer(layer)
QgsProject.instance().removeMapLayer(layer.id())
QgsProject.instance().mapLayers() ## which returns empty dict, all good here

And check there are no files in registry, there is still lock on files: gpkg.wal and gpkg.shm

So i am not able to delete the file with:

os.remove(file)

My only solution is (which is not good, since I would have to call it all the time):

QgsApplication.exitQgs()

After that, I can delete the file. But if I try to start the application again, Python crashes. Any ideas how to release the damn lock?

My script is eg:

def selectByLocation(input, predicate, intersect_lyr, output):
    process = "qgis:extractbylocation"
    params = {
        'INPUT' : input,
        'PREDICATE' : predicate,
        'INTERSECT' : intersect_lyr,
        'OUTPUT' : output
    }
    out = processing.run(process, params)
    return out["OUTPUT"]

def removeLowDepths(raster, threshold, rn, workspace = temp):
    workspace_memory = 'memory:'
    import uuid
    ## cant overwrite file because of lock, so i am using uuid to create many temp files, which i can not delete
    expression = "data < " + str(threshold)
    lowDepthMaskR =  setNull(raster, expression, os.path.join(workspace, "lowDepthMaskR.tif"))
    lowDepthMask = polygonize(lowDepthMaskR, workspace + f'//lowDepthMask_{str(uuid.uuid4())[:8]}.gpkg', epsg)
    fixed = fixGeom(lowDepthMask, workspace_memory +"lowDepthMask_fixed")
    lowDepthMaskSel = selectByLocation(fixed, 0, rn, workspace_memory + "lowDepthMaskSel")
    finalMask = selectByAttribute(lowDepthMaskSel, 'DN', 0, 1, workspace_memory + 'finalMask')

    return finalMask

Edit for comment:

def removeLowDepths(raster, threshold, rn, workspace = temp):
    workspace_memory = 'memory:'
    import uuid, time
    id = str(uuid.uuid4())[:8]
    expression = "data < " + str(threshold)
    lowDepthMaskR =  setNull(raster, expression, os.path.join(workspace, "lowDepthMaskR.tif"))
    lowDepthMask = polygonize(lowDepthMaskR, workspace + f'//lowDepthMask_{id}.gpkg', epsg)

    print(QgsProject.instance().mapLayers()) ## prints {} so empty dict
    del lowDepthMask ## se picture, still locked
    time.sleep(60)
    os.remove(workspace + f'//lowDepthMask_{id}.gpkg')
    return finalMask

enter image description here

Jan Doležal
  • 841
  • 4
  • 12

1 Answers1

1

Unfortunately it seems like this issue can not be solved easily. So for me, it was easier to just switch to another tools and not use QGIS processing (which is a damn shame...)

So I rewrote most processing via GeoPandas, Shapely, GDAL, numpy and it is working as expected. I can delete files just fine now.

def select_by_location(select_from, overlay, output, method = 'intersect', driver='GPKG'):
    if not isinstance(select_from, gpd.GeoDataFrame):
        select_from = gpd.read_file(select_from)
    if not isinstance(overlay, gpd.GeoDataFrame):
        overlay = gpd.read_file(overlay)

    if method.lower() == 'intersect':   
        overlay = gpd.GeoDataFrame({'geometry': overlay['geometry'].values}, crs = overlay.crs)
        selection = gpd.sjoin(select_from, overlay, op='intersects')
        selection = check_geometry(selection)
        if output:
            selection.to_file(output, driver=driver)
        else:
            return selection

    del select_from, overlay
    return output

def check_geometry(input):
    from shapely.wkt import loads
    if not isinstance(input, gpd.GeoDataFrame):
        df = gpd.read_file(input)
    df = input
    df['geometry'] = df['geometry'].astype(str).apply(loads)
    return df
Jan Doležal
  • 841
  • 4
  • 12