Is there a faster process to update one column for all features?

Question

I have written a python script in QGIS which estimates the amount of residents for each building polygon by using census block data and building polygons (about 500 000 buildings in an area of about 900 km2).

The relevant data from the census blocks was merged to the building layer, so only the building layer is used in the script to collect the data. It has the following fields:

| A_building | id_census | pop_census_block |

The steps of the script are:

Sort buildings by id_census
Sum all building footprint areas for each census block
Calculate the areal share and from that the amount of residents for each footprint.
Update 2 attributes of the feature (layer.updateFeature(feat[share, residents]))

The 3rd and 4th step happen in a loop, that iterates over each building polygon. This loop takes two days, which seems to me too long for such a simple operation. Is there a faster way to update the features of a layer, like sending the calculated data as a list?

Here is the relevant section in the script:

# feats is a python list:
feats = [feat for feat in layer.getFeatures(QgsFeatureRequest().setFlags(QgsFeatureRequest.NoGeometry).setSubsetOfAttributes(limit_cols))
]

# sort feats
feats.sort(key=itemgetter(col_census_id))

...

# Get unique census ids
census_ids = []
areas = dict()
for feat in feats:
    census_id = feat[col_census_id]
    if census_id not in census_ids:
        census_ids.append(census_id)
        areas[census_id] = 0

# Sum building area per census block
for i in range(len(feats)):
    # Sum area
    census_id = feats[i][col_census_id]
    areas[census_id] += feats[i][col_bldg_area]

# Start an undo block
layer.beginEditCommand('Calculate share of area')
i, j, k = 0, 0, 0
layer.startEditing()
for feat in feats:
    # Share of area
    share = feat[col_bldg_area] / float(areas[feat[col_census_id]])
    # Add share to feature
    feat[col_share] = share
    # Building population
    feat[col_bldg_pop] = int(feat[col_pop] * feat[col_share])

    layer.updateFeature(feat)

    ...

# End the undo block
layer.endEditCommand()

# Comit changes
layer.commitChanges()
return layer

How big of an area are we talkin' here? Summing polygons for a county could take awhile. One approach I would look at would be to assign all polygons an ID based on the census block - then do your calculations? Looks like your for loop is doing a calculation for each feature which may take longer. — GISHuman, Jul 05 '16 at 13:19
The area is about 900 km2, however summing the polygons only takes a few seconds. My impression is that calling layer.updateFeature() for each polygon takes a lot of time. Do you know how to do some kind of a multi update? — Alexander Gogl, Jul 05 '16 at 14:07
The loop takes about 7 minutes for each 5000 polygons to calculate and update. However after a while the progress slows down and takes hours for each iteration. — Alexander Gogl, Jul 05 '16 at 14:40

score 3 · Accepted Answer · answered Oct 25 '16 at 15:21

Right, there is a way to build a dictionary of changed attributes and pass it directly to the provider's changeAttributeValues method after the whole for loop. The script would look like this:

# feats is a python list:
feats = [feat for feat in layer.getFeatures(QgsFeatureRequest().setFlags(QgsFeatureRequest.NoGeometry).setSubsetOfAttributes(limit_cols))
]

# sort feats
feats.sort(key=itemgetter(col_census_id))

...

# Get unique census ids
census_ids = []
areas = dict()
for feat in feats:
    census_id = feat[col_census_id]
    if census_id not in census_ids:
        census_ids.append(census_id)
        areas[census_id] = 0

# Sum building area per census block
for i in range(len(feats)):
    # Sum area
    census_id = feats[i][col_census_id]
    areas[census_id] += feats[i][col_bldg_area]

# Create the changed attributes dict
attrsMap = {}
col_share_idx = layer.fieldNameIndex( col_share )
col_bldg_pop_idx = layer.fieldNameIndex( col_bldg_pop )
i, j, k = 0, 0, 0
for feat in feats:
    # Share of area
    share = feat[col_bldg_area] / float(areas[feat[col_census_id]])    
    attrsMap[feat.id()] = { 
        col_share_idx: share, 
        col_bldg_pop_idx: int(feat[col_pop] * share) }

    ...

# Save changes
layer.dataProvider().changeAttributeValues( attrsMap )
return layer

It would be interesting to know how long the script takes with this provider-based approach.

Hi Carrillo! Thank you for pointing out the hint. I will try it during the christmas holidays and report how it works out. — Alexander Gogl, Dec 30 '16 at 18:25
On the performance question, there appears to be a substantial penalty to manipulating a layer in edit mode. My experience (as of QGIS 3.22) is changeAttributeValues() scales orders of magnitude more efficiently than calling changeAttributeValue() per feature or calling updateFeature() in a loop and then commitChanges(). For example, for ~80k features on an AMD Zen 3 core at 4.9 GHz with a PCIe 3.0 4x SSD changeAttributeValue() takes around 45 minutes and update+commit several minutes. If the layer's not in edit mode changeAttributeValues(map) is ~2.5 seconds. — Todd West, Jun 23 '22 at 17:55

Is there a faster process to update one column for all features?

1 Answers1

Linked