1

I have a geodatabase with 300 featureclasses with around 4.5 million features in each featureclass. I am updating each featureclass with a value calculated through an algorithm.

The IDLE ide for arcpy script freezes after running for 20 featureclasses. Then I need to forcibly close the script.

How can memory management be done for such a huge dataset?

mkennedy
  • 18,916
  • 2
  • 38
  • 60
anamika singh
  • 19
  • 2
  • If your first 20 feature classes run successfully and the freeze happens on the 21st, what happens when you re-order your feature classes so that the 21st one is processed first? If it freezes, then let's forget about the other 299 and focus this question on that one. – PolyGeo Nov 26 '15 at 11:55
  • Arcpy scripts run-time gets slower when running from one feature class to the next. i.e; time taken for first feature class is not equal to the time taken of the 20th featureclass. How can we clean up memory after running one featureclass sothat for each feature class it runs efficiently. I have already used arcpy.delete_management for deleting the temporary layers. – anamika singh Nov 26 '15 at 12:01
  • 1
    Are you sure that the arcpy script freeze? I had several time the issue that just the outputs freezes but the script is still running in the background. Try to run the script just over 25 featureclasses and check if it comes to an end. You may could split into 15 featureclasse packages. – wittich Nov 26 '15 at 12:14
  • You seemed to avoid my question. Without seeing a code snippet that can be used to reproduce what you are describing, it makes suggesting what to change hard. – PolyGeo Nov 26 '15 at 12:34
  • Yes the script freezed. waited for about 3 hours. the system wasn't responding either. it needed a restart. I think i should try to run on 25 feature classes only and check for the output. @ polygeo the codesnippet we work on is not internet connection so would be not able to post. – anamika singh Nov 26 '15 at 12:54
  • This Q&A may be worth reviewing: http://gis.stackexchange.com/q/13255/115 – PolyGeo Nov 26 '15 at 12:55
  • If you have big table you can use a cursor to update or calculate data. You also need to delete cursor (del cur) at end of loop. print message every 1000 entries. You can already use a scratchworkspace and this : arcpy.env.scratchGDB and purge it with arcpy.Delete_management(scratchWksp) or arcpy.Delete_management(FeatClass1OnScratch) – GeoStoneMarten Nov 26 '15 at 17:03

1 Answers1

1

One thing I would certainly recommend, is to index all fields referenced in Definition Queries of every feature class participating in your batch process.

I have a mixed ModelBuilder / ArcPy tool that regularly churns out 400 feature classes in a single File Geodatabase, the biggest of which was an OpenStreetMap dataset for entire France that resulted in 400 thematic Feature Classes and total file size for the File Geodatabase of 271 GB, and largest Feature Class with almost 42M records. Actually, one of the base Feature Classes that represents the source data for the creation of this large set of Feature Classes, is a point dataset with over 334M records.

This is all running on a modest W7, Core i5 quadcore desktop with 16GB RAM (most of which is actually of no use to ArcGIS for Desktop, since it can't use it like 64 bit ArcGIS Pro).

Making sure all relevant attribute fields had proper indexes, proved to make a real difference on performance. That said, ArcGIS can still appear to "hang" now and then during certain processes, but it rarely fails on this tool despite the huge number of Feature Classes created.

Marco_B
  • 2,582
  • 13
  • 12