Latest Contributions by MarcoBoeringa

Jeff, Will this rasterization option retain the full symbology of existing maps? Is it going to be truly WYSIWYG, except for being rasterized at hopefully high ppi (300+ ppi minimum)? I ask this, because e.g. the high performance engine driving creation of (raster) tiles for web maps, isn't capable of fully retaining symbology, there are some debilitating limitations. This is actually a bit of a headache, as it seems, even if you explicitly choose to build tiles locally instead of in the cloud, you still loose the ability to fully retain detailed topographic styling. Maybe I have missed it while looking over the options, but it seems ESRI actually currently offers no option at all, not even as a simple flattened single layer rasterized image, to retain full high quality cartographic maps in some form of a web mapping service. It would be a pain to see similar limitations in a future PDF rasterization output option, with loss of vital symbology. I really hope for a true WYSIWYG solution. ... View more

Kory, while you're on it, could you please keep in mind, and possibly take this up with the same dev team members, the performance issue and suspision I raised concerning "Tasks" and toolvalidation in Pro: https://community.esri.com/ideas/15362  This may be related, and I have never heard a definitive answer of you whether this was actually being looked and worked on seriously by the dev team, while I think it should. ... View more

Thanks for posting this. This indeed makes it unlikely that you are running into a similar issue as I do. Even though it is not entirely clear at what stage the processing is, the memory usage of Pro in your case is really low, so the issue is likely somewhere else. Since you also run into issues after merging the data, have you actually ever tried to run the Repair Geometry tool against all these input shapefiles? Shapefiles, depending on the source, can be pretty poor and have geometric errors that may not be apparent until you run a tool like yours. I would definitely check the quality by running the Repair Geometry tool (which will also attempt to fix any issues it encounters, you can also run the Check Geometry tool to just check for valid shapes, but this won't fix any issues encountered). ... View more

Fengchao Gu‌ Thanks for the response, but before jumping to the conclusion that this is "the script itself", could you please re-run the script with the same virtual memory settings, but this time actually monitor the Pro allocated memory by either using the ArcGIS Pro Diagnostic Monitor (Use keyboard shortcut Ctrl+Alt+M to open it in a running ArcGIS Pro project), or open the Windows Task Manager's Resource Monitor option (it is on the second Performance TAB of the dialog of the Task Manager). Unless you've truly verified that you do not hit the 111GB alllocatable memory limit you have now set, there is no real garantuee this isn't still a memory issue. Honestly, the highest memory usage I have seen up to now was 64+ GB (see post somewhere down below in this thread)! Especially if you have a second session and some other apps open consuming memory as well, you may still be hitting the same wall... ... View more

Fengchao Gu‌, it would be interesting to hear from you if this indeed solved the issue. Let me know if you got results. It would be another indication that this is not some strange "anomaly" in my system or scripts used, but "as build". ... View more

I have no experience with this specific tool related to portal deployments, but isn't this potentially some rights issue, e.g. that you don't have the proper privileges on the portal to deploy stuff there? I know it is not the error message shown, but it wouldn't be the first time some piece of software returns an unrelated or only marginally related error message upon erroring out. The proper privileges is definitely something to check out I think. ... View more

Patrick Huls‌, that intial link you posted last year "Architecting the ArcGIS Platform: Best Practices" now 404's... That is probably not "best practice" , so I guess you would like to correct that . ... View more

And here is another nice screenshot, a single session of Pro consuming > 64 GB(!) of memory during the export of a complex topographic map layout to PDF format using many layers in the map document, 20 GB in RAM. With my old setup, Pro would have fatally crashed well before reaching this point. Not so now! This is actually the highest value I have seen up to now for a single session. Please note that if you never use more than say two to three dozen layers in Pro on a regular daily basis, you will never see Pro consuming this much memory! ... View more

By the way, I am pretty sure this behavior is not due to some memory leak, but "as build". I think this because of two reasons: - Although allocating a huge amount of memory, Pro is stable and doesn't crash any more, while a memory leak would mean continuous increases in allocated memory and ultimately even with raised virtual memory new crashes when Pro runs out of memory. This hasn't happened at all. - I see Pro properly de-allocating and releasing all of this memory when I close it and when monitoring the Pro allocated memory during this process in Windows Source Control of Task Manager. Again, this isn't behavior you would expect in case of memory leak bug, but looks like proper as designed behavior. Due to the huge amount of allocated memory, closing Pro can be slow, I see it taking up to some 5 minutes to release all of this reserved memory. Only when all of this memory is released, will Pro close definitively. ... View more

Fengchao Gu and @Kory Kramer, I had been battling intermittent but frequent crashes of Pro as well over the last two years. It was really frustrating to see Pro crash every time I pushed it with a large number of layers (in my case to well over 400!). Most frustratingly, sometimes it worked, other times it didn't. Stupidly enough, I only watched Pro's RAM usage in the Task Manager, which never went past let's say 8 GB max (my laptop had 16 GB) and which only shows the memory used for an app actually residing in the physical RAM... hence I never really suspected memory issues... until I finally opened up Source Control from the Task Manager, and discovered Pro was allocating >40(!) GB of memory , almost all of this being stored in Windows virtual memory / swap space, and only a fraction in actual RAM! See the attached image showing only 6,5 GB of allocated memory actually residing in the physical RAM, but >40 GB memory in total allocated for Pro only. This means Pro needs about 75-100 MB for each and every layer it handles in the TOC. This is dramatically more than ArcMap, which can handle the same amount of layers in just 2 GB being a 32 bit app. As a consequence of this realization, I not only bought another 16 GB of RAM for my laptop, but also raised my Windows 10 virtual memory setting to a staggering 80 GB. Since the past two weeks I implemented these changes, all of the crashes have melted away like ice in the desert sun. No more crashes...!!! I seriously have had only one predictable crash since raising the virtual memory to 80GB and raising RAM to 32 GB (so 112 GB of allocatable memory in total), and that was when I deliberately pushed Pro to consume even that gigantic amount of available virtual memory / swap space by opening three concurrent sessions and setting them to work. Eventually, just one instance of Pro crashed when hitting the 112 GB limit, the other two stayed stable and usuable after the crash of the other instance. I therefor strongly recommend you to set at least 50-100 GB of virtual memory in Windows to avoid crashes when processing dozens or hundreds of layers in Pro. It should alleviate your crash issues considerably, if not make them disappear completely, as it did in my case. Pro has been nothing but rock stable since then. Please note that setting that amount of virtual memory is only sensible on SSD storage. Just imagine writing 100 GB to a mechanical hard drive each time Pro needs it as virtual memory... I also think this memory issue is part of the reason of "slowness" in Pro, unless you run Pro and Windows entirely from high end NVMe, the needed work to manage the virtual memory, is likely slowing down Pro in quite some cases. ... View more

I think the first question needs to be: - Do you have at least 100GB free disk space available on an SSD storage for Windows Virtual Memory / swap space? I recently noticed Windows allocating more than 60(!)GB virtual memory while having two sessions of Pro open at a time...Admittedly, I was pushing Pro with projects with over 400 layers in the TOC of a map, but ArcMap opens up similar projects in a few GBs. As I experienced, Pro will crash if you can't take that sort of hit and don't have virtual memory / allocatable free disk space available for this. This will also only properly work with SSD. Imagine writing 50GB to a tradional hard drive each time Pro needs it... ... View more

Kory Kramer‌, just for your record, I got another bug logged through the Dutch branch of ESRI last week that may in fact be related to this one: BUG-000117100 Definition Query gives different results in ArcPro vs ArcMap when a feature class attribute has an index I think the above one may be related since it also concerns NULL values in attributes, and additionally the interpretation of empty strings: '',  and I am pretty sure was introduced in Pro 2.2.2 as well, as I didn't encounter the issue in 2.2.1. For your record also, there is another issue (already in Product Plan if I understood the Dutch branch well) related to empty strings that affects display in Pro dialogs and that got logged a few months ago: BUG-000107874 : In ArcGIS Pro, definition query omits the symbol ' ' if it is surrounded by brackets. This results in validation error for queries such as Include This last one causes queries like: <FIELD A> NOT IN ('') to be displayed like: <FIELD A> NOT IN () in the Definition Query of the layer properties, and thus cause a validation error once you hit "Validate". ... View more

Thomas, Did you ever get a step further with this? Also considering the UC is long past. ... View more