Latest Contributions by MichaelLedwith

‎03-06-2022

Thanks for the explanation Sarmistha (particularly about the internal save), it appears that setting env.cellSize to 10m did the trick. Interestingly enough, setting the cellSize within EnvManager didn't work but setting the cellSize on a separate line cut the time down from 36 hours to less than two.

‎03-04-2022

Update: I converted a single polygon to raster (10m res, 55 X 66 pixels) and reran ZonalStatisticsAsTable. In ArcGIS Pro, it took two (2) seconds! In Python, the process took slightly less than 12 hours. N.B.: it's only ZonalStatisticsAsTable that's giving me trouble. Other arcpy commands run at typical speeds in Python. I've tested this in three different Virtual Desktop environments and on my workstation. The results are all similar. My next step is to completely remove all ESRI traces from one of the VD instances and re-install Pro.

‎03-04-2022

1. The cell size is unchanged, which means that it's 10m (Maximum if Inputs); 2. Percentage: (75%); 3, Saving the output from the Focal Statistics (in this case16 cm resolution, 625 000 X 625 000 pixels) takes hours and hours. The whole reason for using Raster Functions is to greatly increase the speed of the operation; 4. All the inputs have the same defined coordinate system (EPSG: 3006). No settings have been changed or given. Honestly, I'm running the same exact commands. In ArcGIS Pro, the process goes fast, in Python 3.7.11 (IDLE, 64-bit), it goes amazingly slow.

‎03-03-2022

I increased the parallel processing to 75% and re-started the process at 18:42 last evening. It finished at 09:33 this morning. So that's about 15 hours. Definitely better than 36 hours but the improvement might have something to do with less activity on the server during the night. Update: I ran ZonalStatisticsAsTable again this morning in ArcGIS Pro with parallel processing set to 75% and the process time was 1:57:32 (slightly longer).

‎03-02-2022

Well, I certainly want it to be understood that I do appreciate any and all help. I have been using ESRI products (starting with ArcInfo) since the late 80's so I do know a bit about things. What would be great would be an explanation that goes along with any suggestions. For example, why would running the process in a 32-bit IDLE shell speed up the time? Now I don't know what the default parallel processing parameter is for Pro and I didn't change it when I ran the original process. The CPU on my workstation is a Intel i7-6700K with four cores (it's also hyperthreaded, so that makes eight possible cores). I'll re-run the script with a parallel process value of 75% and see what happens. However, I still can't imagine that this parameter could have the effect of changing a process time from 36 hours to less than 2.

‎03-02-2022

I suppose that I haven't explained the situation clearly. I run ZonalStatisticsAsTable from within ArcGIS Pro using the tool. It takes about 1.5 hours. I run ZonalStatisticsAsTable from IDLE (ArcGIS Pro, i.e. python 3.7.11 - 64-bit) using arcpy. It takes about 36 hours. The computer, data sources, environmental settings are exactly the same for both. The only obvious difference is that the inputs are opened in the Table of Contents within ArcGIS Pro (presumably as raster layers). In neither cases are the computer's resources being utilized excessively.

‎03-02-2022

I'll try this, if simply out of curiosity, but I wonder if you could provide a reason as to why reducing the amount of available RAM from 32 GB (64-bit processor) to 4 GB (32-bit processor) would make such a huge difference in time of process. Update: Please inform me as to how I can run arcpy commands for python 3.7.11 in a 32-bit IDLE shell. The Desktop 32-bit IDLE shell doesn't work because I don't have Image Analyst available for Desktop (only Pro). The python.exe file (mentioned above) opens a 64-bit shell.

‎03-02-2022

Well, the AG Pro "script" is: arcpy.ia.ZonalStatisticsAsTable("U2_LPIS_m10_10rast1", "BLOCKID", "Focal Statistics_Extract Bands", r"D:\Projekt\SJV_Blockkontroll2022_XXXXXXX_MILE\Arbete\Produktion\U2\ZonalStatistics\U2_zonalstats.gdb\U2_LPIS_m10_10rast1_textur", "DATA", "MEAN", "CURRENT_SLICE", 90, "AUTO_DETECT") That probably doesn't help you much. The Python script (the last one of many, many that I've tried) is: from arcpy import CheckInExtension, CheckOutExtension from arcpy.ia import ExtractBand, FocalStatistics, NbrRectangle, ZonalStatisticsAsTable from datetime import datetime as dt from os import path print("Starting at {0}".format(dt.now())) CheckOutExtension("ImageAnalyst") proj_mapp = path.join(r"D:\Projekt", "SJV_Blockkontroll2022_XXXXXXX_MILE") arb_dir = path.join(proj_mapp, "Arbete") prod_dir = path.join(arb_dir, "Produktion") mos_dir = path.join(prod_dir, "U2", "Mosaik") mos_gdb = path.join(mos_dir, "U2_mosaic.gdb") in_md = path.join(mos_gdb, "NIR_red_green_mosaic") red_band = ExtractBand(in_md, [1]) textur = FocalStatistics(red_band, NbrRectangle(3, 3, "CELL"), "RANGE") m10 = path.join(arb_dir, "Indata", "U2", "LPIS", "U2_LPIS.gdb", "U2_LPIS_m10_del1") print("Starting Zonal Stats at {0}".format(dt.now())) ZonalStatisticsAsTable(m10, "BLOCKID", textur, path.join(prod_dir, "U2", "ZonalStatistics", "U2_zonalstats.gdb", "U2_LPIS_m10_10rast1_textur_python"), "DATA", "MEAN", "CURRENT_SLICE", 90, "AUTO_DETECT") CheckInExtension("ImageAnalyst") print("Finished at {0}".format(dt.now())) Like I wrote, the script works without any errors and the process time up to running Zonal Statistics as Table can be counted in microseconds. The reason for the excessive amount of path variables has to do with the main script, of which this little snippet is a part. My problem is that it takes about 24 times as long to run it as a Python script than from within ArcGIS Pro.

‎03-02-2022

Hi, I've been pulling out my hair trying to find a solution to this enigma. Zonal Statistics as Table takes less than two hours if I run it using the AG Pro 2.9.2 tool. If I copy the python code and run it in a 64-bit IDLE shell (3.7.11), it takes almost two days. I've tested this on my workstation and three different virtual desktops. All are fairly well equipped Win10 environments with plenty of disk space, RAM, graphic card or graphic support, etc. There's plenty of available cpu- and gpu-resources during the process. Here's the long and short of the situation: 1. The in_zone_data is comprised of slightly more than 4000 polygons that have been converted to a raster with 10 m resolution and a unique ID. The raster has a defined coordinate system and is approximately 5000 X 5000 pixels, unsigned 16-bit; 2. The in_value_raster is the result of two pre-processes. First, the red band is extracted from a RGB-image via Raster Functions (in AG Pro) or arcpy.ia.ExtractBand (in Python). Focal Statistics is then run on the red band via Raster Functions (in AG Pro) or arcpy.ia.FocalStatistics (in Python). The two processes are done almost instantaneously in both AG Pro and Python. The raster has a defined coordinate system, 0.16 m resolution, 625 000 X 625 000 pixels, unsigned 8-bit. In AG Pro, the raster layer source for the Focal Stats raster is on a separate drive (D:\Temp\ArcGISProTemp9836\); 3. Zonal Statistics as Table. Running this tool in AG Pro on the Raster Layers takes about one hour and forty-five minutes. Running the same process (arcpy.ia.ZonalStatisticsAsTable) in Python takes days. 4. I've set up the environments so that they are identical (workspace, scratch, extent, etc). The processing times are the same. 5. I've converted the in_zone_data to a raster layer via arcpy.management.MakeRasterLayer. The processing times are essentially the same. Any explanation? Anyone experience something similar? This snippet is part of a larger program that iterates through tons of different areas so I really need to be able to run this via a script and not manually. No, I cannot provide anyone with the input data but certainly all help appreciated. Thanks!

‎02-24-2022

We naturally appreciate all "attempts" to help... I'll leave that thought there. Seriously though, this problem has ABSOLUTELY NOTHING to do with the specific tool being used. It happens all the time irrespective of the tool or model. It might - might - have something to do with the size of the data set being processed but it just as easily could have something to do with AG Pro's incessant need to map up inordnat amounts of RAM and then not use it or release it.

‎11-04-2021

Thanks for a response... any response - even if it's a few years too late. But seriously, the fact of the matter is that AG Pro attempts to complete some part of the task at hand - a sub-process perhaps. If you're working with a large dataset, AG Pro hangs. This is something I've experienced plenty of time during the past ten years or so. And when I say "plenty" I mean dozens. As for contacting "technical support", good luck with that. We're usually the ones that explain the solutions to ESRI support... 🙂

‎03-01-2021

It's still the same in 2.7.1. I cancelled a SelectByAttribute query (~900,000 objects) last Friday and the cancellation still wasn't finished on Monday morning. The Kill command in ProcessExplorer is the GIS-expert's best friend.

‎10-06-2020

I'm not really sure if you understood the question. It's a general question about pyramid layers. The fact that it is related to a mosaic of aerial photographs is irrelevant. However, I do appreciate the quick response even if it left a lot to be desired.

‎10-06-2020

I am mosaicking together several hundred aerial photos (single band infrared) for different regions of Sweden. I use a script and this works well. It takes between one to two days for the process to complete, including the calculation of statistics and building pyramid layers. The output of the script is an IMG file that can be up to 1 TB in size. This means that an IGE file is also created. I run the script in python 3.6.10 64-bit version. I needed to add a couple of photos to one of the mosaics (686 GB) so I did this manually using AG Pro. The output was a TIF file (don't ask me why, but it probably was by default). No pyramid layers are produced when I use built-in tools. So I decided to build the pyramids manually (i.e. with the Build Pyramids tool in Data Management). I was working locally in a virtual desktop instance (3.6 GHz, 32 GM ram, 64-bit Windows, nVidia GRID p40-8Q) so I just minimized the GUI and forgot about it. When I came back to it a few days later, about 75% of the process was complete. The completion ticked up a few percent per day. By the time the pyramids were built, five minutes shy of eight days had passed. This seems wrong. As mentioned, producing the pyramid layers via script takes less than one day (bigger IMG file). Producing them for a TIF takes eight days. I converted the TIF mosaic to an IMG mosaic and the pyramids were built in less than a day (Take that Cheops!). Is this a known problem with ArcGIS? That large TIF files (presumably big GeoTIFF) have major problems with pyramid layers? Or has it something to do with RRD/RRE vs. OVR files? All help is appreciated!

‎05-13-2020

I'm running an iteration of the results of a region group. There are tens of thousands of rows in the iteration. Each iteration does some stuff and outputs a file that eventually will be part of a much larger mosaic. The scripts runs really fast in the beginning. Then as the number of iterations increase, the processing time becomes longer and longer. For example, the time to run though one thousand iterations (the numbers are per 1000, not cumulative): 0 - 1000: 20 min 1 - 2000: 23 min 2 - 3000: 30 min 3 - 4000: 46 min 4 - 5000: 1:24 5 - 6000: 1:49 6 - 7000: 1:59 7 - 8000: 3:30 8 - 9000: 4:08 9 - 10000: 5:50 So it takes 17 times longer to iterate through lines 9000 - 10000 than it does for lines 0 - 1000. The process ran over a weekend. I checked the directory where the files were being written to and there were tens of thousands of LOCK files. I shut down the IDLE shell, which deleted all of the LOCK files and started up the script again from where I left off. Swoosh! Back up to super fast. I placed a "del row" comment at the end of each iteration but this doesn't seem to release the LOCK file. Anyone know how to get rid of it without stopping the iteration? Or am I missing something else? Is there a way to iterate through tens of thousands of objects, creating separate files without this massive slowdown? I should note that I'm running the script separately on two different computers, as well as two different VD-instances. One each outputs to a directory (tif), the other outputs to a file geodatabase (grid). No difference.

Online Status	Offline
Date Last Visited	‎03-06-2022 04:34 AM

My Ideas

Latest Contributions by MichaelLedwith

Re: Zonal Statistics takes 1.5 hours in AGPro, in Python the same code takes a few days

Re: Zonal Statistics takes 1.5 hours in AGPro, in Python the same code takes a few days

Re: Zonal Statistics takes 1.5 hours in AGPro, in Python the same code takes a few days

Re: Zonal Statistics takes 1.5 hours in AGPro, in Python the same code takes a few days

Re: Zonal Statistics takes 1.5 hours in AGPro, in Python the same code takes a few days

Re: Zonal Statistics takes 1.5 hours in AGPro, in Python the same code takes a few days

Re: Zonal Statistics takes 1.5 hours in AGPro, in Python the same code takes a few days

Re: Zonal Statistics takes 1.5 hours in AGPro, in Python the same code takes a few days

Zonal Statistics takes 1.5 hours in AGPro, in Python the same code takes a few days

Re: Why is ArcGIS PRO so slow to cancel a simple geoprocessing tool ?

Re: Why is ArcGIS PRO so slow to cancel a simple geoprocessing tool ?

Re: Why is ArcGIS PRO so slow to cancel a simple geoprocessing tool ?

Re: Pyramid layers for TIF-files

Pyramid layers for TIF-files

LOCK-file insanity during iteration

Re: Why is ArcGIS PRO so slow to cancel a simple g...

Re: Why is ArcGIS PRO so slow to cancel a simple g...

Re: Why is ArcGIS PRO so slow to cancel a simple g...

Why does AG Pro 2.3 crash every time I move my hea...