Select to view content in your preferred language

ArcGIS Pro Crash while increasing the number of input layers

6995
25
11-09-2018 02:15 AM
FengchaoGu1
Emerging Contributor

I have one feedback about ArcGIS Pro 2.2. It crashes every time when I run a toolbox with increased number of shapefiles (around 56) as input parameters.  The toolbox is created by our team and below is the detailed description.

The function of this toolbox:

Get the input parameters and make attributes selection.

Main arcpy functions:

input = arcpy.GetParameterAsText()

arcpy.SelectLayerByAttribute_management()

Result:

This toolbox can run successfully with flexible numbers of input shapefiles in ArcMap 10.5 or ArcMap 10.6, no matter how many the input layers are.  

G
M
T
Detect language Afrikaans Albanian Amharic Arabic Armenian Azerbaijani Basque Belarusian Bengali Bosnian Bulgarian Catalan Cebuano Chichewa Chinese (Simplified) Chinese (Traditional) Corsican Croatian Czech Danish Dutch English Esperanto Estonian Filipino Finnish French Frisian Galician Georgian German Greek Gujarati Haitian Creole Hausa Hawaiian Hebrew Hindi Hmong Hungarian Icelandic Igbo Indonesian Irish Italian Japanese Javanese Kannada Kazakh Khmer Korean Kurdish Kyrgyz Lao Latin Latvian Lithuanian Luxembourgish Macedonian Malagasy Malay Malayalam Maltese Maori Marathi Mongolian Myanmar (Burmese) Nepali Norwegian Pashto Persian Polish Portuguese Punjabi Romanian Russian Samoan Scots Gaelic Serbian Sesotho Shona Sindhi Sinhala Slovak Slovenian Somali Spanish Sundanese Swahili Swedish Tajik Tamil Telugu Thai Turkish Ukrainian Urdu Uzbek Vietnamese Welsh Xhosa Yiddish Yoruba Zulu
Afrikaans Albanian Amharic Arabic Armenian Azerbaijani Basque Belarusian Bengali Bosnian Bulgarian Catalan Cebuano Chichewa Chinese (Simplified) Chinese (Traditional) Corsican Croatian Czech Danish Dutch English Esperanto Estonian Filipino Finnish French Frisian Galician Georgian German Greek Gujarati Haitian Creole Hausa Hawaiian Hebrew Hindi Hmong Hungarian Icelandic Igbo Indonesian Irish Italian Japanese Javanese Kannada Kazakh Khmer Korean Kurdish Kyrgyz Lao Latin Latvian Lithuanian Luxembourgish Macedonian Malagasy Malay Malayalam Maltese Maori Marathi Mongolian Myanmar (Burmese) Nepali Norwegian Pashto Persian Polish Portuguese Punjabi Romanian Russian Samoan Scots Gaelic Serbian Sesotho Shona Sindhi Sinhala Slovak Slovenian Somali Spanish Sundanese Swahili Swedish Tajik Tamil Telugu Thai Turkish Ukrainian Urdu Uzbek Vietnamese Welsh Xhosa Yiddish Yoruba Zulu
Text-to-speech function is limited to 200 characters

The toolbox can run successfully with less flexible number of input shapefiles (from 1 to 20) in ArcGIS Pro 2.2.

However, when the number of input shapefiles is increased to 56, it crashes and I get the error “ArcGIS Pro has encountered a serious application error and is unable to continue”.

0 Kudos
25 Replies
JonathanFarmer_oldaccount
Deactivated User

Fengchao,

Do you see this same behavior with an out of the box geoprocessing tool in ArcGIS Pro or just this custom toolbox? If it is specific to this custom toolbox and you can narrow down exactly in the Python script where it fails, we could look this over in Support and see if this is an issue on the software end. If it happens with out of the box tools as well, we definitely want to know about it.

Jonathan

FengchaoGu1
Emerging Contributor

Hi Jonathan,

So far it is only this custom toolbox that failed.  I have tried other custom toolboxes created by our team, and they works fine. 

Fengchao 

0 Kudos
KoryKramer
Esri Community Moderator

Can you send (attach) one or two of the dump files associated with this crash?  Find them in C:\Users\<username>\AppData\Local\ESRI\ErrorReports.  You can see the date and time in the filename so that you know it is associated with the workflow you're describing.

That said, any time you experience a crash that is consistently reproducible, please report it directly to technical support - "If you can reproduce the serious application error by a definite series of steps, you should also contact Esri Support or your international distributor to report a bug."

From: Report software errors—ArcGIS Pro | ArcGIS Desktop 

0 Kudos
FengchaoGu1
Emerging Contributor

Hi Kory,

Two of the dump files associated with this crash has been uploaded (See the attachment).  And also, this error has been reported one week ago via "Report software errors" with including my system information in the error report.

Regards,

Fengchao

0 Kudos
KoryKramer
Esri Community Moderator

Thanks.  We have them and will analyze.

0 Kudos
MarcoBoeringa
MVP Alum

Fengchao Gu and @Kory Kramer,

I had been battling intermittent but frequent crashes of Pro as well over the last two years. It was really frustrating to see Pro crash every time I pushed it with a large number of layers (in my case to well over 400!).

Most frustratingly, sometimes it worked, other times it didn't. Stupidly enough, I only watched Pro's RAM usage in the Task Manager, which never went past let's say 8 GB max (my laptop had 16 GB) and which only shows the memory used for an app actually residing in the physical RAM... hence I never really suspected memory issues... until I finally opened up Source Control from the Task Manager, and discovered Pro was allocating >40(!) GB of memory , almost all of this being stored in Windows virtual memory / swap space, and only a fraction in actual RAM! See the attached image showing only 6,5 GB of allocated memory actually residing in the physical RAM, but >40 GB memory in total allocated for Pro only.

This means Pro needs about 75-100 MB for each and every layer it handles in the TOC. This is dramatically more than ArcMap, which can handle the same amount of layers in just 2 GB being a 32 bit app.

As a consequence of this realization, I not only bought another 16 GB of RAM for my laptop, but also raised my Windows 10 virtual memory setting to a staggering 80 GB.

Since the past two weeks I implemented these changes, all of the crashes have melted away like ice in the desert sun. No more crashes...!!!

I seriously have had only one predictable crash since raising the virtual memory to 80GB and raising RAM to 32 GB (so 112 GB of allocatable memory in total), and that was when I deliberately pushed Pro to consume even that gigantic amount of available virtual memory / swap space by opening three concurrent sessions and setting them to work. Eventually, just one instance of Pro crashed when hitting the 112 GB limit, the other two stayed stable and usuable after the crash of the other instance.

I therefor strongly recommend you to set at least 50-100 GB of virtual memory in Windows to avoid crashes when processing dozens or hundreds of layers in Pro. It should alleviate your crash issues considerably, if not make them disappear completely, as it did in my case. Pro has been nothing but rock stable since then.

Please note that setting that amount of virtual memory is only sensible on SSD storage. Just imagine writing 100 GB to a mechanical hard drive each time Pro needs it as virtual memory...

I also think this memory issue is part of the reason of "slowness" in Pro, unless you run Pro and Windows entirely from high end NVMe, the needed work to manage the virtual memory, is likely slowing down Pro in quite some cases.

FengchaoGu1
Emerging Contributor

Thanks Marco! It is quite helpful and detailed. I am going to try to increase the memory.

0 Kudos
MarcoBoeringa
MVP Alum

Fengchao Gu‌, it would be interesting to hear from you if this indeed solved the issue. Let me know if you got results. It would be another indication that this is not some strange "anomaly" in my system or scripts used, but "as build".

FengchaoGu1
Emerging Contributor

@ Marco Boering, Kory Kramer  

I have tried to increase the size of virtual memory to 100GB. But unfortunately, I still get the same crash error. So it seems that because of the script itself. 

0 Kudos
MarcoBoeringa
MVP Alum

Fengchao Gu

Thanks for the response, but before jumping to the conclusion that this is "the script itself", could you please re-run the script with the same virtual memory settings, but this time actually monitor the Pro allocated memory by either using the ArcGIS Pro Diagnostic Monitor (Use keyboard shortcut Ctrl+Alt+M to open it in a running ArcGIS Pro project), or open the Windows Task Manager's Resource Monitor option (it is on the second Performance TAB of the dialog of the Task Manager).

Unless you've truly verified that you do not hit the 111GB alllocatable memory limit you have now set, there is no real garantuee this isn't still a memory issue.

Honestly, the highest memory usage I have seen up to now was 64+ GB (see post somewhere down below in this thread)! Especially if you have a second session and some other apps open consuming memory as well, you may still be hitting the same wall...

0 Kudos