ArcGIS Pro Crash while increasing the number of input layers

[FengchaoGu1]

I have one feedback about ArcGIS Pro 2.2. It crashes every time when I run a toolbox with increased number of shapefiles (around 56) as input parameters.  The toolbox is created by our team and below is the detailed description.

The function of this toolbox:

Get the input parameters and make attributes selection.

Main arcpy functions:

input = arcpy.GetParameterAsText()

arcpy.SelectLayerByAttribute_management()

Result:

This toolbox can run successfully with flexible numbers of input shapefiles in ArcMap 10.5 or ArcMap 10.6, no matter how many the input layers are.  

G

M

T

Detect language Afrikaans Albanian Amharic Arabic Armenian Azerbaijani Basque Belarusian Bengali Bosnian Bulgarian Catalan Cebuano Chichewa Chinese (Simplified) Chinese (Traditional) Corsican Croatian Czech Danish Dutch English Esperanto Estonian Filipino Finnish French Frisian Galician Georgian German Greek Gujarati Haitian Creole Hausa Hawaiian Hebrew Hindi Hmong Hungarian Icelandic Igbo Indonesian Irish Italian Japanese Javanese Kannada Kazakh Khmer Korean Kurdish Kyrgyz Lao Latin Latvian Lithuanian Luxembourgish Macedonian Malagasy Malay Malayalam Maltese Maori Marathi Mongolian Myanmar (Burmese) Nepali Norwegian Pashto Persian Polish Portuguese Punjabi Romanian Russian Samoan Scots Gaelic Serbian Sesotho Shona Sindhi Sinhala Slovak Slovenian Somali Spanish Sundanese Swahili Swedish Tajik Tamil Telugu Thai Turkish Ukrainian Urdu Uzbek Vietnamese Welsh Xhosa Yiddish Yoruba Zulu

Afrikaans Albanian Amharic Arabic Armenian Azerbaijani Basque Belarusian Bengali Bosnian Bulgarian Catalan Cebuano Chichewa Chinese (Simplified) Chinese (Traditional) Corsican Croatian Czech Danish Dutch English Esperanto Estonian Filipino Finnish French Frisian Galician Georgian German Greek Gujarati Haitian Creole Hausa Hawaiian Hebrew Hindi Hmong Hungarian Icelandic Igbo Indonesian Irish Italian Japanese Javanese Kannada Kazakh Khmer Korean Kurdish Kyrgyz Lao Latin Latvian Lithuanian Luxembourgish Macedonian Malagasy Malay Malayalam Maltese Maori Marathi Mongolian Myanmar (Burmese) Nepali Norwegian Pashto Persian Polish Portuguese Punjabi Romanian Russian Samoan Scots Gaelic Serbian Sesotho Shona Sindhi Sinhala Slovak Slovenian Somali Spanish Sundanese Swahili Swedish Tajik Tamil Telugu Thai Turkish Ukrainian Urdu Uzbek Vietnamese Welsh Xhosa Yiddish Yoruba Zulu

Text-to-speech function is limited to 200 characters

Options : History : Feedback : Donate

Close

The toolbox can run successfully with less flexible number of input shapefiles (from 1 to 20) in ArcGIS Pro 2.2.

However, when the number of input shapefiles is increased to 56, it crashes and I get the error “ArcGIS Pro has encountered a serious application error and is unable to continue”.

[FengchaoGu1]

@Marco Boeringa

Here is the screenshot of ArcGIS Pro Diagnostic Monitor and the Performance (See the picture below). The first time I tired to give 50 layers as input but it crashed. And then I merged all the 50 layers into one layer. try it again but it still crashes. 

And the total size of the merged file(.shp) is 364KB.

G

M

T

Detect languageAfrikaansAlbanianAmharicArabicArmenianAzerbaijaniBasqueBelarusianBengaliBosnianBulgarianCatalanCebuanoChichewaChinese (Simplified)Chinese (Traditional)CorsicanCroatianCzechDanishDutchEnglishEsperantoEstonianFilipinoFinnishFrenchFrisianGalicianGeorgianGermanGreekGujaratiHaitian CreoleHausaHawaiianHebrewHindiHmongHungarianIcelandicIgboIndonesianIrishItalianJapaneseJavaneseKannadaKazakhKhmerKoreanKurdishKyrgyzLaoLatinLatvianLithuanianLuxembourgishMacedonianMalagasyMalayMalayalamMalteseMaoriMarathiMongolianMyanmar (Burmese)NepaliNorwegianPashtoPersianPolishPortuguesePunjabiRomanianRussianSamoanScots GaelicSerbianSesothoShonaSindhiSinhalaSlovakSlovenianSomaliSpanishSundaneseSwahiliSwedishTajikTamilTeluguThaiTurkishUkrainianUrduUzbekVietnameseWelshXhosaYiddishYorubaZulu

AfrikaansAlbanianAmharicArabicArmenianAzerbaijaniBasqueBelarusianBengaliBosnianBulgarianCatalanCebuanoChichewaChinese (Simplified)Chinese (Traditional)CorsicanCroatianCzechDanishDutchEnglishEsperantoEstonianFilipinoFinnishFrenchFrisianGalicianGeorgianGermanGreekGujaratiHaitian CreoleHausaHawaiianHebrewHindiHmongHungarianIcelandicIgboIndonesianIrishItalianJapaneseJavaneseKannadaKazakhKhmerKoreanKurdishKyrgyzLaoLatinLatvianLithuanianLuxembourgishMacedonianMalagasyMalayMalayalamMalteseMaoriMarathiMongolianMyanmar (Burmese)NepaliNorwegianPashtoPersianPolishPortuguesePunjabiRomanianRussianSamoanScots GaelicSerbianSesothoShonaSindhiSinhalaSlovakSlovenianSomaliSpanishSundaneseSwahiliSwedishTajikTamilTeluguThaiTurkishUkrainianUrduUzbekVietnameseWelshXhosaYiddishYorubaZulu

Text-to-speech function is limited to 200 characters

Options : History : Feedback : Donate

Close

[MarcoBoeringa]

Thanks for posting this. This indeed makes it unlikely that you are running into a similar issue as I do. Even though it is not entirely clear at what stage the processing is, the memory usage of Pro in your case is really low, so the issue is likely somewhere else.

Since you also run into issues after merging the data, have you actually ever tried to run the Repair Geometry tool against all these input shapefiles? Shapefiles, depending on the source, can be pretty poor and have geometric errors that may not be apparent until you run a tool like yours. I would definitely check the quality by running the Repair Geometry tool (which will also attempt to fix any issues it encounters, you can also run the Check Geometry tool to just check for valid shapes, but this won't fix any issues encountered).

[FengchaoGu1]

Thanks for the suggestion.

I have tried to use Check Geometry to check all my input files and also use Repair Geometry to repair to make sure the quality is good.  But I still face this crash problem. The problem is still the same, less number of input files, this tool can handle, but more... It crashes. The crash point is around 40-45 number of layers.

[MarcoBoeringa]

By the way, I am pretty sure this behavior is not due to some memory leak, but "as build". I think this because of two reasons:

- Although allocating a huge amount of memory, Pro is stable and doesn't crash any more, while a memory leak would mean continuous increases in allocated memory and ultimately even with raised virtual memory new crashes when Pro runs out of memory. This hasn't happened at all.

- I see Pro properly de-allocating and releasing all of this memory when I close it and when monitoring the Pro allocated memory during this process in Windows Source Control of Task Manager. Again, this isn't behavior you would expect in case of memory leak bug, but looks like proper as designed behavior.

Due to the huge amount of allocated memory, closing Pro can be slow, I see it taking up to some 5 minutes to release all of this reserved memory. Only when all of this memory is released, will Pro close definitively.

[MarcoBoeringa]

And here is another nice screenshot, a single session of Pro consuming > 64 GB(!) of memory during the export of a complex topographic map layout to PDF format using many layers in the map document, 20 GB in RAM. With my old setup, Pro would have fatally crashed well before reaching this point. Not so now! This is actually the highest value I have seen up to now for a single session.

Please note that if you never use more than say two to three dozen layers in Pro on a regular daily basis, you will never see Pro consuming this much memory!

[ThomasColson]

Are you running the toolbox as a batch? Your py shows 1 input for feature class, yet your screenshot shows many.