Latest Contributions by KenHartling

After taking a look at Mary's case I was unable to reproduce the performance issues using her data and model. She was able to work around the issue by modifying the order of operations a bit in the model, however, I would guess that there is something on the system being used that is causing the issue. I recommended following some of the troubleshooting issues here when possible. Mary, thanks for helping us attempt to investigate what you were seeing.    Ken ... View more

Hi Matthew, It sounds like you have reached some memory limitations.  http://desktop.arcgis.com/en/arcmap/latest/tools/data-management-toolbox/dissolve.htm The availability of physical memory may limit the amount (and complexity) of input features that can be processed and dissolved into a single output feature. This limitation could cause an error to occur, as the dissolve process may require more memory than is available. To prevent this, Dissolve may divide and process the input features using an adaptive tiling algorithm. To determine the features that have been tiled, run the Frequency tool on the result of this tool, specifying the same fields used in the dissolve process for the Frequency Field(s) parameter. Any record with a frequency value of 2 has been tiled. Tile boundaries are preserved in the output features to prevent the creation of features that are too large to be used by ArcGIS. Caution: Running Dissolve on the output of a previous dissolve run will rarely reduce the number of features in the output when the original processing divided and processed the inputs using adaptive tiling. The maximum size of any output feature is determined by the amount of available memory at run time; therefore, output containing tiles is an indicator that dissolving any further with the available resources would cause an out-of-memory situation or result in a feature that is unusable. Additionally, running the Dissolve tool a second time on output that was created this way may result in very slow performance for little to no gain and may cause an unexpected failure. When you see this type of output in ArcMap, please make sure to try the same process using Background 64bit ( http://desktop.arcgis.com/en/arcmap/latest/analyze/executing-tools/64bit-background.htm ). Pro being 64bit should also help but it appears there may have been issues there as well.  Let me know if you would be willing to share your data with me and I'll send you info on how to share it with us securely. Thx. Ken ... View more

Hello Mary, Would I be able to get the model and data from you to investigate? If so, I'll contact you with info on how to get it to us securely. Thx. Ken ... View more

Is this a stand alone script run outside of the ArcGIS UI's? If so... your script has no selection so it will append everything. In the script you will need to create a layer based on the in_layer, make a selection (SelectLayerByAttribute, SelectLayerByLocation, etc) and then run Append. ... View more

Have you reported this to your support contact? Please do, we'd like to see the issue so we can investigate. Thx. Ken ... View more

Alessandro, Have you reported this to your support contact?  If you submit a support case we'll be able to investigate and if necessary put in a fix. Thanks. Ken ... View more

Please help us by sending us the case you see 'Shirley' crashing so we can take a look.  Either via your support contact or we can arrange for you to send it to me directly.  It would also be good to know your machine specs. I'd love to see your '2 days for us to intersect 30 million points with 1k drive time polygons' case as well.  2 days seems a little too long to me unless the machine it is being run on is pretty slow or doesn't have adequate resources. Pro has had the ability to run most of the overlay tools in parallel for some time (set arcpy.env.parallelProcessingFactor = 100 before running the tool). If you have a machine that can handle it, you may be able to get more performance out of many overlay operations. Thanks. ... View more

Over the past several years there have been many advancements in processing large data on the Desktop.  In ArcMap we've added 64bit background, ArcGIS Pro is a native 64bit application.  We've been modifying our products to take more and more advantage of the 64bit world and the additional resources it allows us to use when processing data.   I would highly encouraging everyone that needs to process any data of size or complexity to give these 64bit offerings a try... and at the same time I would discourage dicing up features prior to processing if at all possible. Running Dice on your data should be a last resort. Mervyn, if you do run into a process that won't complete using a 64bit offering (lots of stuff 'should' fail when run using 32bit applications) please let me know. Feel free to contact me with the data and case personally. Thanks! Ken ... View more

Dice should only be used on features that your system cannot handle at all (can't edit, can't draw, etc. See Dicing Godzillas (features with too many vertices) | ArcGIS Blog  ). Chopping up features like this could actually slow some processes down and cause other issues and require additional post processing. If you have a specific case where this is required to get through the data efficiently could I see it? If you can, I'll send you a private message on how to get it to me efficiently. Thanks. Ken ... View more

8GB of RAM isn't a whole lot if your data contains large complex features with complex interactions... number of features may also be a factor depending on the amount of resources your machine has. You can get a better understanding of this by reading this old blog: Be successful overlaying large, complex datasets in Geoprocessing | ArcGIS Blog  That said, can you share the data and the exact GP tool call (with all parameters) that is taking a long time? If you can, let me know here and I'll send you a private message on how you can get it to me efficiently. Thanks. Ken (ESRI GP Product Engineer) ... View more

Looking at Dave's data I found that there are many places in the data where there are thousands and thousands of overlapping polygons. This is a particular challenge for topological tools. The underlying engine, the Topology Engine (TE) is designed to create a topological fabric across the entire dataset and then work out all the topological relationships between all the features. With thousands upon thousands of polygons interacting over most of the dataset in question there will be a fair resource requirement in order to complete the operation. On 32bit Dave's case would page badly. At the same time the engine gets bogged down in figuring out how every polygon interacts with each and every other polygon. In ArcGIS Pro we developed new tools that can be of help. The Pairwise Tools (Pro only). For data like Dave has created we released the PairwiseDissolve tool. The PairwiseDissolve tool with Dave's data took just under 1 minute on my machine as apposed to many hours using the Dissolve tool.  The PairwiseDissolve tool is a geometry operation that is more lightweight than the TE and it runs in parallel mode taking advantage of all the cores on your machine. Data precision in both the PairwiseDissolve and TE generated output is extremely good.   Although the DM/Generalization/Dissolve tool is extremely efficient for most cases you throw at it, cases like this with extreme overlap can cause it to run slow and in 32bit environments failure is possible. For these cases there is often nothing that will get a Dissolve operation using the DM/Generalization/Dissolve tool to perform anywhere near the PairwiseDissolve tool. We built the pairwise tools to take care of these scenarios.  If you are performing any operations of size or complexity I would highly recommend you only run using python 64bit (which comes with Pro) with more than the minimum system requirements for Pro. For overlay operations investigate running the Pairwise tools or Analysis tools that run in parallel mode.  In ArcGIS Pro 2.1 (coming soon) many of the tools under the Analysis/Overlay toolbox support a parallel option for area-area, area-line and area-pnt. Pro 2.0 overlay tools only support parallel for area-area overlays. Note: For the Pairwise tools, be sure to look very carefully at the doc for PairwiseIntersect… output is quite different than Analysis/Overlay/Intersect.   As a reminder... Pro can be installed and run on the same machine (provided its windows 64bit) at the same time as the other ArcGIS applications.  More info - http://pro.arcgis.com/en/pro-app/get-started/about-licensing.htm    To run python scripts make sure to run your scripts with the appropriate python install from the ArcGIS product you with to run them with (ArcMap, Pro...) - http://pro.arcgis.com/en/pro-app/arcpy/get-started/installing-python-for-arcgis-pro.htm ... View more

Would I be able to get a copy of the data to see what the real problem may be? Thanks. Ken (ESRI GP Product Engineer) ... View more

I have confirmed this buffer process is too large/complex for 32bit processing. Using 64bit ArcGIS Desktop Background Processing, ArcGIS Server (64-bit) and ArcGIS Pro works fine. ... View more

It is a separate install (see link I provided) and will show up in Windows Programs and Features as "ArcGIS Desktop Background Geoprocessing 10.4 (64-bit)". You can control whether you use Background processing by following the steps here - Foreground and background processing—Help | ArcGIS Desktop  ... View more