topic Re: ArcGIS Pro Crash while increasing the number of input layers in ArcGIS Pro Questions

ArcGIS Pro Crash while increasing the number of input layers

FengchaoGu1 — Fri, 09 Nov 2018 10:15:09 GMT

I have one feedback about ArcGIS Pro 2.2. It crashes every time when I run a toolbox with increased number of shapefiles (around 56) as input parameters. The toolbox is created by our team and below is the detailed description.

The function of this toolbox:

Get the input parameters and make attributes selection.

Main arcpy functions:

input = arcpy.GetParameterAsText()

arcpy.SelectLayerByAttribute_management()

Result:

This toolbox can run successfully with flexible numbers of input shapefiles in ArcMap 10.5 or ArcMap 10.6, no matter how many the input layers are.

The toolbox can run successfully with less flexible number of input shapefiles (from 1 to 20) in ArcGIS Pro 2.2.

However, when the number of input shapefiles is increased to 56, it crashes and I get the error “ArcGIS Pro has encountered a serious application error and is unable to continue”.

Re: ArcGIS Pro Crash while increasing the number of input layers

ThomasColson — Sat, 10 Nov 2018 14:21:59 GMT

Can you share your Python script and toolbox? Not a lot to troubleshoot using just screen shots in a Python error.

Re: ArcGIS Pro Crash while increasing the number of input layers

ThomasColson — Mon, 12 Nov 2018 14:50:04 GMT

Wow....that's a tech support case for sure.In the Pro version, what happens when you move

calc_dr_selection_temp.xlsx

to another directory with no spaces or special characters and where your computer login has "Full Control" to the directory?

Re: ArcGIS Pro Crash while increasing the number of input layers

JonathanFarmer_oldaccount — Mon, 12 Nov 2018 20:03:19 GMT

Fengchao,

Do you see this same behavior with an out of the box geoprocessing tool in ArcGIS Pro or just this custom toolbox? If it is specific to this custom toolbox and you can narrow down exactly in the Python script where it fails, we could look this over in Support and see if this is an issue on the software end. If it happens with out of the box tools as well, we definitely want to know about it.

Jonathan

Re: ArcGIS Pro Crash while increasing the number of input layers

KoryKramer — Mon, 12 Nov 2018 20:59:13 GMT

Can you send (attach) one or two of the dump files associated with this crash? Find them in C:\Users\<username>\AppData\Local\ESRI\ErrorReports. You can see the date and time in the filename so that you know it is associated with the workflow you're describing.

That said, any time you experience a crash that is consistently reproducible, please report it directly to technical support - "If you can reproduce the serious application error by a definite series of steps, you should also contact Esri Support or your international distributor to report a bug."

From: Report software errors—ArcGIS Pro | ArcGIS Desktop

Re: ArcGIS Pro Crash while increasing the number of input layers

FengchaoGu1 — Tue, 13 Nov 2018 09:08:16 GMT

Hi I tried to move it to another directory with no spaces or special characters but it does not work. I got the same crash. And since it works in ArcGIS Pro with less number of input, and it only crash when there are more than 50 input. So I do not really think it is about this .xlsx template.

Re: ArcGIS Pro Crash while increasing the number of input layers

FengchaoGu1 — Tue, 13 Nov 2018 09:12:42 GMT

Hi Jonathan,

So far it is only this custom toolbox that failed. I have tried other custom toolboxes created by our team, and they works fine.

Fengchao

Re: ArcGIS Pro Crash while increasing the number of input layers

FengchaoGu1 — Tue, 13 Nov 2018 09:29:03 GMT

Hi Kory,

Two of the dump files associated with this crash has been uploaded (See the attachment). And also, this error has been reported one week ago via "Report software errors" with including my system information in the error report.

Regards,

Fengchao

Re: ArcGIS Pro Crash while increasing the number of input layers

ThomasColson — Tue, 13 Nov 2018 14:46:24 GMT

Are you running the toolbox as a batch? Your py shows 1 input for feature class, yet your screenshot shows many.

Re: ArcGIS Pro Crash while increasing the number of input layers

KoryKramer — Tue, 13 Nov 2018 16:40:55 GMT

Thanks. We have them and will analyze.

Re: ArcGIS Pro Crash while increasing the number of input layers

MarcoBoeringa — Wed, 14 Nov 2018 22:20:41 GMT

Fengchao Gu and @Kory Kramer,

I had been battling intermittent but frequent crashes of Pro as well over the last two years. It was really frustrating to see Pro crash every time I pushed it with a large number of layers (in my case to well over 400!).

Most frustratingly, sometimes it worked, other times it didn't. Stupidly enough, I only watched Pro's RAM usage in the Task Manager, which never went past let's say 8 GB max (my laptop had 16 GB) and which only shows the memory used for an app actually residing in the physical RAM... hence I never really suspected memory issues... until I finally opened up Source Control from the Task Manager, and discovered Pro was allocating >40(!) GB of memory , almost all of this being stored in Windows virtual memory / swap space, and only a fraction in actual RAM! See the attached image showing only 6,5 GB of allocated memory actually residing in the physical RAM, but >40 GB memory in total allocated for Pro only.

This means Pro needs about 75-100 MB for each and every layer it handles in the TOC. This is dramatically more than ArcMap, which can handle the same amount of layers in just 2 GB being a 32 bit app.

As a consequence of this realization, I not only bought another 16 GB of RAM for my laptop, but also raised my Windows 10 virtual memory setting to a staggering 80 GB.

Since the past two weeks I implemented these changes, all of the crashes have melted away like ice in the desert sun. No more crashes...!!!

I seriously have had only one predictable crash since raising the virtual memory to 80GB and raising RAM to 32 GB (so 112 GB of allocatable memory in total), and that was when I deliberately pushed Pro to consume even that gigantic amount of available virtual memory / swap space by opening three concurrent sessions and setting them to work. Eventually, just one instance of Pro crashed when hitting the 112 GB limit, the other two stayed stable and usuable after the crash of the other instance.

I therefor strongly recommend you to set at least 50-100 GB of virtual memory in Windows to avoid crashes when processing dozens or hundreds of layers in Pro. It should alleviate your crash issues considerably, if not make them disappear completely, as it did in my case. Pro has been nothing but rock stable since then.

Please note that setting that amount of virtual memory is only sensible on SSD storage. Just imagine writing 100 GB to a mechanical hard drive each time Pro needs it as virtual memory...

I also think this memory issue is part of the reason of "slowness" in Pro, unless you run Pro and Windows entirely from high end NVMe, the needed work to manage the virtual memory, is likely slowing down Pro in quite some cases.

Re: ArcGIS Pro Crash while increasing the number of input layers

MarcoBoeringa — Thu, 15 Nov 2018 00:49:12 GMT

By the way, I am pretty sure this behavior is not due to some memory leak, but "as build". I think this because of two reasons:

- Although allocating a huge amount of memory, Pro is stable and doesn't crash any more, while a memory leak would mean continuous increases in allocated memory and ultimately even with raised virtual memory new crashes when Pro runs out of memory. This hasn't happened at all.

- I see Pro properly de-allocating and releasing all of this memory when I close it and when monitoring the Pro allocated memory during this process in Windows Source Control of Task Manager. Again, this isn't behavior you would expect in case of memory leak bug, but looks like proper as designed behavior.

Due to the huge amount of allocated memory, closing Pro can be slow, I see it taking up to some 5 minutes to release all of this reserved memory. Only when all of this memory is released, will Pro close definitively.

Re: ArcGIS Pro Crash while increasing the number of input layers

MarcoBoeringa — Thu, 15 Nov 2018 06:39:02 GMT

And here is another nice screenshot, a single session of Pro consuming > 64 GB(!) of memory during the export of a complex topographic map layout to PDF format using many layers in the map document, 20 GB in RAM. With my old setup, Pro would have fatally crashed well before reaching this point. Not so now! This is actually the highest value I have seen up to now for a single session.

Please note that if you never use more than say two to three dozen layers in Pro on a regular daily basis, you will never see Pro consuming this much memory!

Re: ArcGIS Pro Crash while increasing the number of input layers

FengchaoGu1 — Thu, 15 Nov 2018 13:18:22 GMT

Thanks Marco! It is quite helpful and detailed. I am going to try to increase the memory.

Re: ArcGIS Pro Crash while increasing the number of input layers

MarcoBoeringa — Mon, 19 Nov 2018 06:27:28 GMT

Fengchao Gu‌, it would be interesting to hear from you if this indeed solved the issue. Let me know if you got results. It would be another indication that this is not some strange "anomaly" in my system or scripts used, but "as build".

Re: ArcGIS Pro Crash while increasing the number of input layers

FengchaoGu1 — Mon, 26 Nov 2018 15:11:49 GMT

@ Marco Boering, Kory Kramer

I have tried to increase the size of virtual memory to 100GB. But unfortunately, I still get the same crash error. So it seems that because of the script itself.

Re: ArcGIS Pro Crash while increasing the number of input layers

MarcoBoeringa — Mon, 26 Nov 2018 17:01:12 GMT

Fengchao Gu‌

Thanks for the response, but before jumping to the conclusion that this is "the script itself", could you please re-run the script with the same virtual memory settings, but this time actually monitor the Pro allocated memory by either using the ArcGIS Pro Diagnostic Monitor (Use keyboard shortcut Ctrl+Alt+M to open it in a running ArcGIS Pro project), or open the Windows Task Manager's Resource Monitor option (it is on the second Performance TAB of the dialog of the Task Manager).

Unless you've truly verified that you do not hit the 111GB alllocatable memory limit you have now set, there is no real garantuee this isn't still a memory issue.

Honestly, the highest memory usage I have seen up to now was 64+ GB (see post somewhere down below in this thread)! Especially if you have a second session and some other apps open consuming memory as well, you may still be hitting the same wall...

Re: ArcGIS Pro Crash while increasing the number of input layers

FengchaoGu1 — Thu, 29 Nov 2018 09:00:15 GMT

@Marco Boeringa

Here is the screenshot of ArcGIS Pro Diagnostic Monitor and the Performance (See the picture below). The first time I tired to give 50 layers as input but it crashed. And then I merged all the 50 layers into one layer. try it again but it still crashes.

And the total size of the merged file(.shp) is 364KB.

Re: ArcGIS Pro Crash while increasing the number of input layers

MarcoBoeringa — Thu, 29 Nov 2018 18:19:44 GMT

Thanks for posting this. This indeed makes it unlikely that you are running into a similar issue as I do. Even though it is not entirely clear at what stage the processing is, the memory usage of Pro in your case is really low, so the issue is likely somewhere else.

Since you also run into issues after merging the data, have you actually ever tried to run the Repair Geometry tool against all these input shapefiles? Shapefiles, depending on the source, can be pretty poor and have geometric errors that may not be apparent until you run a tool like yours. I would definitely check the quality by running the Repair Geometry tool (which will also attempt to fix any issues it encounters, you can also run the Check Geometry tool to just check for valid shapes, but this won't fix any issues encountered).

Re: ArcGIS Pro Crash while increasing the number of input layers

ColinZwicker — Thu, 29 Nov 2018 23:18:55 GMT

Fengchao,

Instead of building a large OR list for your linkID's

where += ' OR "LINK_ID" = {0}'.format(int(linkid))

Can you try making the whereclause in the form of an in list?, e.g., IN (ID1, ID2, ID3, N)? Ideally this IN clause would contain no duplicated ID's (so perhaps collect them in a list and then only add the unique values).

re: "I have tried to increase the size of virtual memory to 100GB. But unfortunately, I still get the same crash error. So it seems that because of the script itself. "

As the thread has somewhat devised this issue isnt about heap memory but about stack memory and the usage of that stack while attempting to parse a whereclause with a growing number of logical operators.

Thanks,

Colin