Field Calculator and Calculate Geometry EXTREMELY SLOW!!!

[ThomasEmery]

We manage land records for a county that contains 191,532 parcels. Within the attribute table there are 4 fields which require us the calculate the geometry and 3 fields which which require the use of the field calculator. This took less than 5 minutes per field in 9.3.1 but takes over 20 minutes in 10.
This could potentially cause us to go back to 9.3.1 until something is resolved.


What in the world is going on? Has anyone encountered this and found a solution?

Thanks

[MattGoodman]

I'm running a field calculation concatenating two fields, in an SDE geodatabase point file with 62,000 records.  Elapsed time was  2 hours 17 minutes 31 seconds. 

There is no join. 

I saw some forum discussion recommending using VBA parsing to speed up field calcs when there is a join involved - does that mean that when there is not a join involved that Python parsing is faster? 

Any tips/tricks for helping this process run a little faster?

Thanks,  -Matt

[KenHartling]

Our testing in house has not shown an overall slow down in performance between ArcGIS 9.3.1 and 10.0 for the CalculateField tool. With selection sets we've seen improvements in performance at 10.0.  We have seen some slowdown when updating field values in an entire feature class (no selection set) when there are tens of millions of records to be updated.  We are investigating this for 10.1. If the feature class contains less than 10 million records we've seen improved performance when updating a field value for an entire feature class.

If any of you on this thread have specific examples and data you can share with us we would love to be able to have them so we can investigate what you are reporting.  There may be something different about your case that we have not considered in our testing.  Any help you can give us in figuring this out would be greatly appreciated.  If you are unable to share your case and data through the forum, you can:
1. Submit a technical support incident.
2. Use our ftp site to upload the data and let me know it is there. (I can provide instructions)

When providing us with instructions on how to reproduce the issue you are seeing, please provide:
- The exact parameters you are using in the CalculateField tool when you encounter the issues you have reported.
- The patch level of ArcGIS you have applied.
- If you have comparison numbers between 9.3.1 and 10.0 please provide us with your performance numbers for each version using your test case.
- Please provide the machine specifications for the machines that were used for each version of ArcGIS in your testing.

A few things of note for the CalculateField tool ...

We've addressed a number of performance issues for CalculateField in 10.0 SP3 (coming soon). Once SP3 is available, you may want to apply it and retest to see if what you are reporting has been helped by these fixes.

A fix for the PYTHON parser option slowness when compared to VB is planned to be ported back to 10.0 SP4 (release date TBD).

10.1 (release date TBD) contains a very substantial improvement to working with selection sets in GP tools.  CalculateField performance when updating field values using a selection set has been improved dramatically in 10.1.  Unfortunately, these changes cannot be ported back to earlier releases.

The ArcSDE team is investigating slow performance when the CalculateField tool is used against ArcSDE.  Something to be aware of in our testing of ArcSDE with the CalculateField tool is direct connect connections appear substantially slower than when using application server connections for very large updates using the CalculateField tool.

Thanks for your help. 
Ken
ESRI - Senior Geoprocessing Product Engineer

[AnnHowland]

I also am experiencing the same slowness with ArcMAP 10.  I have a direct connection SDE feature class layer with about 70,000 records that I have to use the Toolbox to join this address layer to the update parcels layer to update three fields.  The Field Calculation used to take about 30 - 45 minutes for all three fields to recalculate WITH the join.   Now it takes ONE field calculation 3-4 HOURS to recalculate. So I spend nine to twelve hours of work time waiting on my computer to recalculate.  Because there are 70,000 records, I can't run anything else in the meantime or it takes even longer.  What a waste of time and productivity.

[KenHartling]

Would you be able to share your data (as a fgdb) and the exact steps of setting up the join and running the calculateField tool so we can investigate?  If you don't want to share it through the forum just let me know and I can send you instructions for uploading it all to our secure ftp site.

Thanks, Ken
ESRI
Senior GP Product Engineer

I also am experiencing the same slowness with ArcMAP 10.  I have a direct connection SDE feature class layer with about 70,000 records that I have to use the Toolbox to join this address layer to the update parcels layer to update three fields.  The Field Calculation used to take about 30 - 45 minutes for all three fields to recalculate WITH the join.   Now it takes ONE field calculation 3-4 HOURS to recalculate. So I spend nine to twelve hours of work time waiting on my computer to recalculate.  Because there are 70,000 records, I can't run anything else in the meantime or it takes even longer.  What a waste of time and productivity.

[NathenHarp]

I am also having trouble with Add Joins and Calculate fields in Arc10.0. Have developed very simple model and script, no changes between database env., .dbf, .gdb. But there is a problem with processing record counts. The model/script is successful with very small datasets, i.e., 50-100.

I ran the model/script in Arc9.3.1 and at ~600,000 records, ~20 minutes. When seperating out the processes and calculating the field in ArcMap, took < 20 secs.

Has this bug been resolved in Arc10.1?

[CarolWickenheiser]

Would you be able to share your data (as a fgdb) and the exact steps of setting up the join and running the calculateField tool so we can investigate?  If you don't want to share it through the forum just let me know and I can send you instructions for uploading it all to our secure ftp site.

Thanks, Ken
ESRI
Senior GP Product Engineer

Ken, I would be glad to share our data. We are using the parcel fabric, calculating from a joined field, in ArcMap version 10.0,  SDE version 10.0,  SQL 2008 R2. I have exported the fabric into a file gdb to upload to ESRI for a separate issue so it is ready to go if you would like me to send it to you. Please send me instructions. Thank you. - Carol

[StephaniePatterson]

I am having the same problem. I have a parcel geodatabase in a personal geodatabase v10 and it now takes hours to calculate the geometry for the acreage. This is really slowing down our update process.

Stephanie

[JohnFell]

We are using ArcSDE 10.0 SP 4, SQL 2008 R2, direct connections and using python to execute the CalculateField tool. It takes about 12 minutes to calculate about 70 rows within a child version of default in a parcel fabric. I was curious if there was a performance enhancement request pending or a workaround to this behavior. Thanks.

[MathewCoyle]

We are using ArcSDE 10.0 SP 4, SQL 2008 R2, direct connections and using python to execute the CalculateField tool. It takes about 12 minutes to calculate about 70 rows within a child version of default in a parcel fabric. I was curious if there was a performance enhancement request pending or a workaround to this behavior. Thanks.

Does using a cursor to calculate the geometry work better? Something like this?

import arcpy

layer = "some_layer_or_path_to_fc"
desc = arcpy.Describe(layer)
shapeField = desc.shapeFieldName
field = "Area_field_to_update"

ucurs = arcpy.UpdateCursor(layer)
for row in ucurs:
    area_set = row.getValue(shapeField).area
    row.setValue(field,area_set)
    ucurs.updateRow(row)

[JohnFell]

mzcoyle,

After applying your solution the script tool was able to execute successfully in a fraction of the time it took to calculate the selected features. Thank you for the suggestion!