1

I want to delete all points inside a polygon layer:

import arcpy
import time

arcpy.env.workspace = r"E:\DensityMaps\DensityMapsTest1.gdb"
arcpy.env.overwriteOutput = True

start_time = time.time()

arcpy.Erase_analysis("Test_Points_Project", "Europe_coastline_NoLakesRivers", "PointsPolyErase1")

print "Erase ended "+("--- %s seconds ---" % (time.time() - start_time))

This lasts 36,9 seconds

Then I read this post saying that the process can be dramatically optimized using in_memory. I give it a try:

import arcpy
import time

arcpy.env.overwriteOutput = True
start_time = time.time()

#Copy points to in_memory
inFeature = r"E:\DensityMaps\DensityMapsTest1.gdb\Test_Points_Project"
memoryFeature = "in_memory\PointsMemory"

arcpy.CopyFeatures_management(inFeature, memoryFeature)

#Erase points that are inside polygons
arcpy.Erase_analysis(memoryFeature, r"E:\DensityMaps\DensityMapsTest1.gdb\Europe_coastline_NoLakesRivers", "PointsMemory")

#Copy points back to directory
arcpy.CopyFeatures_management("PointsMemory", r"E:\DensityMaps\DensityMapsTest1.gdb\PointsPolyErase")

print "Erase ended "+("--- %s seconds ---" % (time.time() - start_time))

This lasts 38.4 seconds, 1,5 seconds slower than the previous script. I am obviously doing something wrong. Very wrong.

It's very important to make the process faster because the original point file contains 22 millions points (1,5 GB). If I don't manage to do it fast the operation can last about 24 hours.

Here are the files if you want to test

Community
  • 1
Manuel Frias
  • 497
  • 4
  • 11
  • 4
    Twenty-two million points is far too large for fast operation anywhere. Memory-based techniques are designed for far smaller datasets. Deletion is a very expensive operation -- Don't use it. Just select the features you want to keep, and copy them out instead. – Vince Sep 08 '15 at 13:28
  • That was only 15 secs. Good improvement, thanks. Is there any way to make it even faster? – Manuel Frias Sep 08 '15 at 14:18
  • 2
    No. Only you know how many features per second you're seeing, but it's already probably maxxed-out in terms of throughput. Working with large and very large datasets takes a very different mindset. – Vince Sep 08 '15 at 14:24

1 Answers1

4

There are a number of issues here. First off, 22 million features is a large or possibly very large table. These terms have a very specific meaning in terms of database operation; basically, they mean that you can't ever expect anything to happen quickly, and must constantly evaluate all aspects of everything you do with them to be as efficient as possible.

While in_memory feature classes can be used to improve some operations, they are essentially a long list in memory, and there are operations which do not lend themselves to efficient operation. When you add the overhead of copying into memory (building the list), and copying back out, it's rare that large or very large tables will receive any benefit from in_memory operation (the processing must be significant to outweigh the setup cost).

Deletion is not an efficient operation. The rule on DELETE for large tables is "Don't do it" (the rule for experts is "Don't do it yet"). As discussed in the comments, there isn't any reason to delete when you can select the features you want to keep, and copy them out instead. Doing so will be slow (depending on the number of remaining features), but it's your only option (beyond using an attribute index to ignore the features you don't need).

Unfortunately, working with large tables often requires a willingness to redefine success. It helps if you try to evaluate operations in terms of processing rate instead of duration. When you're getting processing rates at hundreds of thousands of features per second, you need to be able to exclaim "Not bad!" and move on to the next bottleneck.

Vince
  • 20,017
  • 15
  • 45
  • 64