topic Re: Feature Class to Feature Class tool extremely slow Arc10 SP2 in Geoprocessing Questions

Feature Class to Feature Class tool extremely slow Arc10 SP2

ReneRubalcava — Fri, 27 May 2011 19:52:31 GMT

Wasn't sure if this should go here or Python forum, but I was able to replicate this issue in Model Builder, so I figured this was a good spot.

So this week we decided to test SP2 on ArcMap 10. My PC is usually the guinea pig for service packs and we typically have no problems.

Light testing this week had been fine, until I tried to run a Python script we use to export data from our SDE to some shapefiles for a legacy application. Typically, this script would take approximately 10 minutes to run (I keep logs of every run). Today, a single FC to FC task was taking up to 30 minutes. I do this a couple of times and killed the script after an hour.

So I copied the exact same script to another pc with ArcMap 10 (no SP2) and ran the same script. It took about 6 minutes on that machine.

I tested the FC to FC tool in ArcToolbox and again the process was taking a ridiculous amount of time to export a single FeatureClass.

Here is an old log from my machine (pre-SP2) for reference.

2011-05-03 07:05:51.416000 | Create SDE Connection file and export SDE Features to Shapefiles
2011-05-03 07:06:00.709000 | SDE Connection created successfully
2011-05-03 07:06:00.709000 | SDE Connection file is ready
2011-05-03 07:06:00.709000 | Export SDE Nodes to shp
2011-05-03 07:06:00.709000 | Export SDE Pipes to shp
2011-05-03 07:08:33.498000 | Delete original locarc shapefile
2011-05-03 07:08:33.701000 | Rename new locarc shapefile to replace old shapefile
2011-05-03 07:08:34.029000 | Export SDE local lines to shp
2011-05-03 07:12:38.404000 | Create SDE Connection file and export SDE Features to Shapefiles
2011-05-03 07:12:46.118000 | SDE Connection created successfully
2011-05-03 07:12:46.118000 | SDE Connection file is ready
2011-05-03 07:12:46.118000 | Export SDE Nodes to shp
2011-05-03 07:12:46.118000 | Export SDE Pipes to shp
2011-05-03 07:12:46.118000 | Export SDE local lines to shp
2011-05-03 07:15:02.437000 | Delete original locnode shapefile
2011-05-03 07:15:02.593000 | Rename new locnode shapefile to replace old shapefile
2011-05-03 07:15:02.859000 | Export SDE local lines to shp
2011-05-03 07:15:02.859000 | Export of SDE to SHP completed successfully

This script exports four SDE features to shapefiles as well as performs some other calculations. The only place that seems to get hung up in the script is the Feature Class to Feature Class operation.

My script can't even finish with SP2 install, so I don't have a current log file.
I'll try to let it run tonight to see if I can finish saving a log file with accurate times.

Has anyone else experienced similar issues with SP2? Is there a bug list somewhere maybe I can refer to?

For reference, my current ArcMap/ArcCatalog build is build 3200 with SP2.
My previous ArcMap/ArcCatalog build (and current build of all other ArcMap installs in office) is build 2800.

Thank you.

Re: Feature Class to Feature Class tool extremely slow Arc10 SP2

ReneRubalcava — Fri, 27 May 2011 20:26:49 GMT

I don't know if this matters at all, but I was running the FC to FC tool right now via ArcToolbox in ArcCatalog and 40 minutes (approx 42% done I think, tough to tell in that statusbar) in I decided to cancel it. Cancelling took about 5 minutes and I was monitoring the output file sizes before and after cancelling.

sbn, shp, shx file sizes did not change at all.
The dbf was 21.5mb before cancelling, and in 5 minutes from cancelling went to 91.5mb.
Even though, I right-clicked -> cancel in the Result Window, I still got a success message and the Result Window says it was a success and not a cancel. Don't know if it just does that.

I attached screenshots for reference, including my ArcToolbox env settings (which should be default, I have not touched these).

Re: Feature Class to Feature Class tool extremely slow Arc10 SP2

ReneRubalcava — Tue, 31 May 2011 12:57:14 GMT

For reference, I placed a counter on my script and left it running Friday night.
The result

2011-05-28 19:13:59.655000 | Total time to execute = 1 day, 4:24:41.896000

Took over a day.
I am going to request that we try to update another machine with SP2 to verify.
I was kind of hoping others were having similar problems before we try to contact our ESRI Support.

I've run the Fc to Fc tool on network and local shapefiles and they work just fine.
When I run the tool against an SDE feature, it is really slow.
If I monitor the statusbar %, it gets to 99%, then starts over again at 0% and seems to get stuck in a loop. Because I can cancel and it will finish without errors and the data seems to be intact.

Re: Feature Class to Feature Class tool extremely slow Arc10 SP2

GeorgeStrebel — Thu, 02 Jun 2011 15:48:27 GMT

For reference, I placed a counter on my script and left it running Friday night.
The result
2011-05-28 19:13:59.655000 | Total time to execute = 1 day, 4:24:41.896000
Took over a day.
I am going to request that we try to update another machine with SP2 to verify.
I was kind of hoping others were having similar problems before we try to contact our ESRI Support.

I've run the Fc to Fc tool on network and local shapefiles and they work just fine.
When I run the tool against an SDE feature, it is really slow.
If I monitor the statusbar %, it gets to 99%, then starts over again at 0% and seems to get stuck in a loop. Because I can cancel and it will finish without errors and the data seems to be intact.

We are having a similar problem running python models in 10 SP2 and have gotten the same behaviors with similar testing. Your documentation of the issue is much more in-depth and detailed then anything we have so far. Please submit a problem report for this issue, SP2 "fixes" must have created a new bug. Thanks for getting out ahead of this on the forums.

Re: Feature Class to Feature Class tool extremely slow Arc10 SP2

ReneRubalcava — Mon, 11 Jul 2011 19:01:38 GMT

Just thought I would post an update to this issue we were having.
Still waiting for a bug id, but it would appear the issue occurs when trying to use the feature class to feature class tool with a SDE dataset that has a geometric network when using SP2. For now, workaround is to export the data via ArcMap.

Re: Feature Class to Feature Class tool extremely slow Arc10 SP2

NidhinKarthikeyan — Tue, 12 Jul 2011 21:22:53 GMT

This behaviour is a Bug in ArcGIS Desktop 10 SP2. [Exporting a Feature class associated with a network dataset to a shapefile using FC2FC extremely slow]

The behavior can be reproduced with File Geodatabase / Personal Geodatabase / ArcSDE Geodatabase.

The export process runs in a loop and once we hit cancel, we can find that the export is complete.[Confirm the number of features in attribute table]

Workaround:
Add the feature class to ArcMap > browse for the feature class in table of contents > right click on feature class > data > export data.

Fixed in ArcGIS Desktop 10.1 🙂

Re: Feature Class to Feature Class tool extremely slow Arc10 SP2

KayAnderson — Wed, 02 Nov 2011 12:08:37 GMT

I am having the same issue. I killed the session and the export was complete with the correct number of records in the attribute table. However, there are additional fields being created at the end of the table that are concatenated from existing field names. For example I have a field named PRE_DIR that is valid but at the end of the table a new field has been created called PRE_D_DESC. There are no attributes in the field. This appears to be the reason it is taking so long. See attached.

Re: Feature Class to Feature Class tool extremely slow Arc10 SP2

AlexJohnson — Tue, 08 Nov 2011 19:44:47 GMT

This problem bit us after our upgrade to SP2 last month. We export 30+ featureclasses from ArcSDE periodically, so we created a Python script and a set of tools to automate the process. At SP1 it took about 15-20 mintues to run. After SP2, it tool over 6 hours. On ESRI's suggestion we updated to SP3, but saw very little change.

We worked with ESRI tech support, isolating the problem to the FeatureClass-to-FeatureClass tool. We tested it with a non-geometric network featureclass (~29,000 points) and a geometric network featureclass (~24,000 points). Results were as follows:

10.0 SP3
Non-net featureclass: 17 min 30 sec
Net featureclass: 51 min 10 sec

10.0 SP1
Non-net featureclass: 18 sec
Net featureclass: 22 sec

The geoprocessing output and my notes:

SP3 Exports:
--Address FC, no geometric network, 29,088 points, 11/4/11
Start Time: Fri Nov 04 13:49:30 2011
Succeeded at Fri Nov 04 14:07:00 2011 (Elapsed Time: 17 minutes 30 seconds)
--Ran through the progress bar twice - once really fast, then very slowly.

--Sewer Service Connection FC, geometric network, 23,882 points, 11/4/11
Start Time: Fri Nov 04 14:09:48 2011
Succeeded at Fri Nov 04 15:00:58 2011 (Elapsed Time: 51 minutes 10 seconds)
--Ran through progress bar at least five times.

SP1 Exports:
--Address FC, no geometric network, 29,088 points, 11/4/11
Start Time: Fri Nov 04 19:33:55 2011
Succeeded at Fri Nov 04 19:34:13 2011 (Elapsed Time: 18.00 seconds)
--Ran through progress bar once, fast.

--Sewer Service Connection FC, geometric network, 23,882 points, 11/4/11
Start Time: Fri Nov 04 19:36:25 2011
WARNING 000645: Output format does not support GlobalIDs. Field GlobalID will not be converted to output.
Succeeded at Fri Nov 04 19:36:47 2011 (Elapsed Time: 22.00 seconds)
--Ran through progress bar once.

Obviously, SP3 (and SP2) are causing a big slowdown. One symptom we observed was that at SP2 and SP3, the progress bar in the featureclass-to-featureclass tool would run multiple times.

Finally, ESRI was able to duplicate the problem, and they were unable to correct it. Their suggested work-around was to keep on client computer at SP1 for our exports.

ESRI tech support logged a bug, if you're interested in tracking it:
'NIM075066: FeatureClassToFeatureClass_conversion performance times are slower'.

Re: Feature Class to Feature Class tool extremely slow Arc10 SP2

KenCarrier — Fri, 11 May 2012 16:44:26 GMT

I just checked with Esri, it still will exist in SP4 and SP5(July 2012) and also in 10.1 pre-release.

Re: Feature Class to Feature Class tool extremely slow Arc10 SP2

ReneRubalcava — Fri, 11 May 2012 17:41:20 GMT

We were told this issue would be fully fixed with 10.1, so I'm hoping that is still the case. We've held off an all SP upgrades above SP1 because of this bug.

Re: Feature Class to Feature Class tool extremely slow Arc10 SP2

DrewFlater — Mon, 14 May 2012 16:42:41 GMT

I'm very sorry for the difficulty everyone is having with the slow-downs for the Feature Class To Feature Class (Conversion) tool. As has been indicated above, Esri knows about a performance issue with this tool that was introduced in 10.0 SP2.

In that service pack, new functionality was added so that when geodatabase data with attribute domains is converted to shapefile (which does not support domains), a new field is added to the output shapefile to maintain the domain description information. The short story of the problem is this additional step of adding the domain description text causes large increases in processing time.

This issue is being worked on in 10.0 Service Pack 5, and will provide significant improvements in performance. In addition to the performance improvements, in the 10.1 release, a new geoprocessing environment is added so that you can control if domain description information should be maintained in the output.

People on this board have replied with some good workarounds -- you can use the ArcMap Export function, or use the Copy Features (Management) geoprocessing tool to copy your geodatabase data to shapefile -- neither of these methods will maintain domain descriptions so performance will be superior.

Good luck to everyone,
Drew -Esri

Re: Feature Class to Feature Class tool extremely slow Arc10 SP2

JohnGargiulo — Sat, 23 Jan 2016 15:14:16 GMT

I am having a similar issue in 10.3 release. When my block group data is joined to a tabular data set, the "feature class to feature class" and "Copy Features" tools in Model Builder take at least 10x as long to run compared to a manual Data-Export function. I understand 217k rows is a decent amount of data, but if the manual process takes less than 5 minutes, why would the model builder process take an hour? Also running into memory issues on some iterations. Anyone else experiencing this in later versions?