topic Re: Different results when running arcpy outside ArcMap? in Python Questions

Different results when running arcpy outside ArcMap?

JamesRamm — Thu, 09 Oct 2014 11:38:22 GMT

I am trying to run ZonalStatisticsAsTable using python.

I have a feature class in a FGDB and a raster within an SDE database (SQL Server).

If I load these into layers within arcmap, open the python console (within arcmap) and execute:

arcpy.CheckOutExtension("Spatial")
arcpy.sa.ZonalStatisticsAsTable(r"Site Boundary\Site", "Site", "sde_jbacfm.JBAGISEDITOR.River09Q1000D", "TempTable")

I get a table of results.

If I run this code, from a python console outside of arcmap:

import arcpy
arcpy.CheckOutExtension("Spatial")

# In these filepaths I have purposefully replaced the full path with '...' for clarity. This is just for this example
arcpy.env.workspace = r"C:\...\Output.gdb" # This is an empty geodatabase initiated with a WGS84 projection

input_feat = r"C:\...\SitePolygon.gdb\Site Boundary\Site"
raster = r"C:\...\connection.sde\sde_jbacfm.JBAGISEDITOR.River09Q1000D"
output = "TempTable"

arcpy.sa.ZonalStatisticsAsTable(input_feat, "Site", raster, output)

This works but I get an entirely different result for the statistics.

I am trying to work out what is going on? The only thing I can think of is that the projection of each input (and the output database) is different, but I had understood that arcpy would handle different projections automatically?

In any case, I would still expect to get the same result using python through arcmap as I did not reproject anything there either.

Re: Different results when running arcpy outside ArcMap?

DanPatterson_Retired — Thu, 09 Oct 2014 12:18:25 GMT

first this line should be put in raw format arcpy.env.workspace = "C:\...\Output.gdb"

I wouldn't assume anything about...nor would I rely upon...projections on the fly. You can easily fix that by establishing them explicitly

Re: Different results when running arcpy outside ArcMap?

JamesRamm — Thu, 09 Oct 2014 13:39:02 GMT

The line is in raw format; just an error in my editing for this forum.

Projections on the fly or not...it doesnt explain why it would behave differently within arcmap than outside?

The two should be the same (right?), so should both give the same result, right or wrong.

Re: Different results when running arcpy outside ArcMap?

curtvprice — Thu, 09 Oct 2014 17:43:38 GMT

When I see a difference in tool output inside and outside ArcMap (especially raster tools) my first place to check is the geoprocessing environment (extent, cellsize, output coordinate system, etc). If the environment settings are different in the ArcMap document than the environment defaults that are used when you run a python script standalone, you are likely to see different results.

> The only thing I can think of is that the projection of each input (and the output database) is different, but I had understood that arcpy would handle different projections automatically?

Yes, but its automatic choice will depend on your geoprocessing settings and the input datasets. This varies by tool. In this situation, you probably want to explicitly set the spatial reference. Zonal Statistics analysis is area-weighted so you should specify an equal-area (projected) coordinate system rather than just letting arcpy geoprocessing guess based in your inputs.

The easiest way to do do this in Python is to use the EPSG code:

arcpy.env.outputCoordinateSystem = arcpy.SpatialReference(102039) # Albers equal area

Re: Different results when running arcpy outside ArcMap?

JamesRamm — Tue, 14 Oct 2014 12:48:13 GMT

The env information is exactly the same, apart from the output workspace - however, these are both just file geodatabases with the same parameters.

env is mostly the same because the parameters are not set (i.e. extent, cellsize, output coordinate system are all blank in both arcmap and stand-alone).

I would've thought that in this case though, arcpy would always choose the same parameters for any settings it requires which are not user-provided, as the inputs are exactly the same whether it is arcmap or stand-alone.

Re: Different results when running arcpy outside ArcMap?

JamesRamm — Tue, 14 Oct 2014 12:54:19 GMT

You are right:

Setting

arcpy.env.outputCoordinateSystem

ensures the same answer whether from within arcmap or stand alone, however it is a different answer than that gained when run from within arcmap without

arcpy.env.outputCoordinateSystem.

We have decided that the latter case is the right answer, so I'm still trying to figure out what has gone wrong.

Re: Different results when running arcpy outside ArcMap?

curtvprice — Tue, 14 Oct 2014 20:18:17 GMT

Sounds like I had your answer. The ArcMap defaults clearly aren't the same as what you expected.

When I do raster processing I try to project the data (Project Raster, Resample) instead of project on the fly, because then I have full control of how the resampling works.

Explicitly controlling the raster processing environment (cell size, snap raster, etc) makes sure you are comparing apples with apples.

Re: Different results when running arcpy outside ArcMap?

JamesRamm — Wed, 15 Oct 2014 11:18:17 GMT

Yes I have it working by projecting the input feature to the same projection as the raster to extract stats from.

It does seem a bit non-intuitive that arcpy would opaquely alter settings from within arcmap and not without.

Re: Different results when running arcpy outside ArcMap?

curtvprice — Wed, 15 Oct 2014 17:10:03 GMT

> It does seem a bit non-intuitive that arcpy would opaquely alter settings from within arcmap and not without.

When running from ArcMap, there are settings the user can set in the geoprocessing environment dialog, and in modelbuilder -- and also if you call your tool from a python script, it may have environment settings before your tool is called. To support this, the tool behavior must be affected by the GP environment set in the application. So if you care (like in this case, with zonal stats) you need to explicitly set the env in your tool. Your script documentation should inform the user which geoprocessing settings your tool will honor.

Desktop Help 10.2: Environment Levels and Hierarchy

Re: Different results when running arcpy outside ArcMap?

JamesRamm — Mon, 20 Oct 2014 09:15:55 GMT

Seems like I've not fully resolved this issue as it appears that trying to project two of the same feature, but with different coordinate systems, to the same coordinate system gives slightly different results.

Importing two features which are the same but have different SRS's to arcmap correctly displays them (i.e. on top of each other).

Reprojecting both to a 3rd coordinate system outside of arcmap and then importing these new features now shows that they are not in the same position.

I'm wondering what other environment settings are controlling this.

Re: Different results when running arcpy outside ArcMap?

curtvprice — Thu, 23 Oct 2014 02:33:16 GMT

This sounds like a different issue. Please start a new thread with more detail (i.e. which spatial references, data types, even some test data).

Re: Different results when running arcpy outside ArcMap?

giuseppemolinario — Thu, 08 Jun 2017 00:25:14 GMT

In any case, I would still expect to get the same result using python through arcmap as I did not reproject anything there either.

I published a peer reviewed paper in 2015, based on tabulated results that were obtained using a python script that used the arcpy method zonal histogram. I had no environments set there, other than snap to raster. The area shapefile and the raster, were in the same projection. Recently,having to check those results again, I used zonal histogram in the ArcMap GUI and, using the same exact input layers, and also not setting any environments other than snap to raster, I got drastically different results.

How is this not a bug? The answer to the OP should be simple and clear: sorry - we'll fix this.

arcpy/ArcMap should not have opaque settings that produce different results, when the same tool is used and all the input layers and settings are equal. Period.

It took me several hours of debugging to figure out the issue - the default (and OPAQUE) environment settings in zonal histogram when used from the arcmap GUI analyze the data at a coarser resolution than the inputs, to save computational time. Setting the environment setting to match the pixel size of the raster being analyzed, makes the tool perform as expected, and as it should be as default, obviously, and therefore matches the output results i get when using zonal histogram on the same exact layers in an arcpy script.

gdal next time?

Re: Different results when running arcpy outside ArcMap?

curtvprice — Thu, 08 Jun 2017 03:53:40 GMT

It's only fair to note in this thread that the help for this tool Zonal Histogram—Help | ArcGIS Desktop and Zonal Statistics as Table—Help | ArcGIS Desktop both have a bullet item warning about this behavior. (No doubt in response to user input in the form of support tickets!)

It is recommended to only use rasters as the zone input, as it offers you greater control over the vector-to-raster conversion. This will help ensure you consistently get the expected results.

I am the author of a tool that does Zonal stats and I have it avoid this problem by defaulting the cell size (and snap) to the raster dataset resolution as you suggest. Unless the environment cell size has been set, in which case it honors the env cell size, resampling the value raster, but warning the user.

More on environments: Setting geoprocessing environments—Help | ArcGIS Desktop

Re: Different results when running arcpy outside ArcMap?

DanPatterson_Retired — Thu, 08 Jun 2017 06:11:57 GMT

At least in PRO they have put the Environments tab at the top, front and center instead of being buried in the bottom right corner of the dialog

Small steps... I would love to have a non-removable popup when settings haven't been explicitly set OR checked. Would make the classroom go a lot smoother

Re: Different results when running arcpy outside ArcMap?

curtvprice — Thu, 08 Jun 2017 09:16:00 GMT

I totally agree with giuseppe molinario that the environments are quite a bit below the hood, and that sometimes it can be absolutely maddening when you get results you didn't expect because of environment settings set at the application level inside a dialog that is not even included as a button on the ArcMap toolbar! (Personally I use Customize mode to add an environments button next to the Python button.)

I also am missing a button you can push in the environments dialog to "reset all environments" to defaults. Except, the question is asked: which defaults, as the current settings are saved with the map document. Something that is supposed to be a convenience can become a curse in a hurry.

I hope the links I offered were in a spirit of helpfulness to readers of this thread, and definitely not in disrespect to anyone frustrated with the platform, which is very powerful like a knife with many blades that can cut you. Been there many times in the last 30 years, I get it. Trust but verify.