Geoprocessing: Doesn't utilise Processor or Memory effectively

Anonymous User · ‎09-26-2012

Original User: Playa

I've noticed that a very large amount of the geoprocessing tools found within the ArcToolbox doesn't effectively utilise the processor or memory within my PC. A simple task of projecting a point feature class only utilises 10% of the processor and 90 MB of memory. The process has been running for more than 10min just to project two feature classes. This is a common trend among all the Geoprocessing Tools found within ArcToolBox. The utilisation of PC resources needs to be drastically improved within ArcGIS and especially the Geoprocessing environment.

PeterWilson · ‎09-26-2012

I've noticed that a very large amount of the geoprocessing tools found within the ArcToolbox doesn't effectively utilise the processor or memory within my PC. A simple task of projecting a point feature class only utilises 10% of the processor and 90 MB of memory. The process has been running for more than 10min just to project two feature classes. This is a common trend among all the Geoprocessing Tools found within ArcToolBox. The utilisation of PC resources needs to be drastically improved within ArcGIS and especially the Geoprocessing environment.

The projection of the two feature classes took 1hr19min!

Regards

Anonymous User · ‎09-26-2012

Original User: ken2472

Hi,

I've asked the Product Engineer for the project tool to contact you about your specific project operation. They should be able to point you to some of the specific details about how Projecting data works that may explain what you are seeing. They also will be able to evaluate your data and specific Project operation to determine if there is a performance issue with your case.

In regards to your general statements:
"a very large amount of the geoprocessing tools found within the ArcToolbox doesn't effectively utilise the processor or memory"
"This is a common trend among all the Geoprocessing Tools found within ArcToolBox"

Could you be more specific? We try to design our tools to perform all operations as efficiently as possible. If you feel some operations you are running are not completing in a timely manner, please send these cases to us, with data if possible, for investigation (through this forum, through support or feel free to send them to me directly). Knowing what your hardware specifications are for the machine you are running on will be important for the investigation.

As far as how efficiently they are using your CPU's or RAM, we also would need to see the specific cases you are worried about in order to determine if there is anything we can do.

The overlay tools are good example of a set of tools that maximize their use of memory when processing data. You would not see them exhibit the behavior you mention in your post. Our overlay tools attempt to processes as much data as possible, all at once, within the bounds of the available resources in order to provide a topologically correct output across the entire dataset. Please see the following doc and blog for more details:
Tiled processing of large datasets
Be successful overlaying large, complex datasets in Geoprocessing

However, overlay operations are suited for this type of resource utilization due to the type of output that is desired. There are many other operations (different tools) where it would be inefficient to take this approach.

Thanks,
Ken
Senior Geoprocessing Product Engineer

ChrisSnyder · ‎09-26-2012

"a very large amount of the geoprocessing tools found within the ArcToolbox doesn't effectively utilise the processor or memory... This is a common trend among all the Geoprocessing Tools found within ArcToolBox"

There are some notable exceptions, but I have generally found the oposite to be true. Most gp processes occupy 100% of a single CPU, and more or less, efficiently manage memory.

In your specific case, it's probably more likely that there is something awry in your work flow. Could it be that you are reading/writting this data over a slow network connection?

Anonymous User · ‎09-27-2012

Original User: Playa

Hi,

I've asked the Product Engineer for the project tool to contact you about your specific project operation. They should be able to point you to some of the specific details about how Projecting data works that may explain what you are seeing. They also will be able to evaluate your data and specific Project operation to determine if there is a performance issue with your case.

In regards to your general statements:
"a very large amount of the geoprocessing tools found within the ArcToolbox doesn't effectively utilise the processor or memory"
"This is a common trend among all the Geoprocessing Tools found within ArcToolBox"

Could you be more specific? We try to design our tools to perform all operations as efficiently as possible. If you feel some operations you are running are not completing in a timely manner, please send these cases to us, with data if possible, for investigation (through this forum, through support or feel free to send them to me directly). Knowing what your hardware specifications are for the machine you are running on will be important for the investigation.

As far as how efficiently they are using your CPU's or RAM, we also would need to see the specific cases you are worried about in order to determine if there is anything we can do.

The overlay tools are good example of a set of tools that maximize their use of memory when processing data. You would not see them exhibit the behavior you mention in your post. Our overlay tools attempt to processes as much data as possible, all at once, within the bounds of the available resources in order to provide a topologically correct output across the entire dataset. Please see the following doc and blog for more details:
Tiled processing of large datasets
Be successful overlaying large, complex datasets in Geoprocessing

However, overlay operations are suited for this type of resource utilization due to the type of output that is desired. There are many other operations (different tools) where it would be inefficient to take this approach.

Thanks,
Ken
Senior Geoprocessing Product Engineer

Dear Ken

I've emailed Nobbir Ahmed

The specs of my PC, version of ArcGIS, a sample of the data and print screens of the processing time.

The "Project Feature" and "Append" are generally very slow and utilise on a small percentage of the processor and memory. I took the appended feature class and used "Export To CAD" and it very quickly utilised most of the processor and as well as 3GB of memory.

Regards