topic Re: Script are changed by ESRI with g_ESRI_variable_1, g_ESRI_variable_2 in Python Questions

Script are changed by ESRI with g_ESRI_variable_1, g_ESRI_variable_2

simoxu — Fri, 10 Dec 2021 19:55:46 GMT

It's a surprise to me that ArcGIS Pro changes the python code in my script tool when I pack the project and upload the ppkx to ArcGIS Online.

Here is the first a few lines of code for one script tool in the unpacked ppkx, please pay attention to the codes between "Esri start of added imports" and "Esri end of added variables"

# Esri start of added imports
import sys, os, arcpy
# Esri end of added imports

# Esri start of added variables
g_ESRI_variable_1 = "EventID = '{}'"
g_ESRI_variable_2 = os.path.join(arcpy.env.scriptWorkspace,'..\\..\\..\\Users\\simoxu
\\AppData\\Local\\Esri\\ArcGISPro\\Staging\\SharingProcesses\\0038
\\RDA Tools for ArcGIS Pro\\p20\\current_checkout_attachments')
# Esri end of added variables

 #-------------------------------------------------------------------------------
# Name:        rda_checkout
# Purpose:     checkout the assessment for a specific event and map it
#
# Author:      simoxu
# Created:     27/07/2018
#-------------------------------------------------------------------------------
from arcpy import da
import os,sys
import shutil
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

In one place in my original code, I have the following line:

checkout_image_path= os.path.join(homefolder,"current_checkout_attachments")‍‍

But it was replaced with the following code, which is causing fatal error for the tool itself.

checkout_image_path= os.path.join(homefolder,g_ESRI_variable_2)‍

It's quite strange that ArcGIS Pro packaging tool will change my code without informing me --- I only found out this when I shared the project package with others and then was told the tools could not run properly.

Is it only me? Any advice?

I am using ArcGIS Pro 2.2.1, by the way.

Thanks.

Re: Script are changed by ESRI with g_ESRI_variable_1, g_ESRI_variable_2

DanPatterson_Retired — Sun, 12 Aug 2018 02:28:49 GMT

Package Project—Data Management toolbox | ArcGIS Desktop

it seems some of your paths are to your folder and even if you are sharing internally, I doubt that anyone has access to it. You should consolidate all the data, scripts etc into the folder structure that you are packaging.

Connections—such as folder, server, database and workflow—will only be included in an internal package. These items are removed if the package is created to be shared outside your organization.

Re: Script are changed by ESRI with g_ESRI_variable_1, g_ESRI_variable_2

simoxu — Mon, 13 Aug 2018 01:06:36 GMT

Thanks Dan.

That's the problem --- ESRI-added variable points to my folder which no one else can access. My question is, why does the Package Project tool do this? it seems the tool is confused by my code, somehow.

I understand the Package Project tool will analyze my custom tool before consolidating the required resources, so I manually used the Analyze Tools for Pro to analyze my script, and there are no related issues reported in result.

As you can see in my code, I am creating a path using os.path.join() function, the second parameter is a string (folder name) which is replaced by an ESRI-added variable ---- what? I am using a string, a CONSTANT, why changed it to a variable?

Re: Script are changed by ESRI with g_ESRI_variable_1, g_ESRI_variable_2

DanPatterson_Retired — Mon, 13 Aug 2018 01:59:11 GMT

os.path.join(arcpy.env.scriptWorkspace,'..\\..\\..\\Users\\simoxu ….. isn't that you?

Your workspace should be in the same folder as your whole project

c:\MyProjects\

\my.aprx

\mydata.gdb

\myscratch.gdb

\myscripts

\the_script.py

Re: Script are changed by ESRI with g_ESRI_variable_1, g_ESRI_variable_2

ModyBuchbinder — Mon, 13 Aug 2018 04:46:27 GMT

The packaging is really making the tool ready for server. It should keep paths that are registered in the server and try to replace others since the server does not have access to them. In your case I think it try to pack everything since the package should be self contained and should not use any other paths. Sometime the only way with server is to edit the result script. You can try to unzip, fix the script and zip again but this is not supported!!

Have Fun

Re: Script are changed by ESRI with g_ESRI_variable_1, g_ESRI_variable_2

simoxu — Mon, 13 Aug 2018 04:49:23 GMT

I am pretty sure all the workspaces I use in the project are all under the same project folder.

The absolute path pointing to "..\\users\\simoxu...." were added in the front of my code by ArcGIS Pro Package Project tool / function, I am quite puzzled by this behavior.

Because it is pointing to "C:\Users\<USERPROFILE>\AppData\Local\ESRI\ArcGISPro\Staging\SharingProcesses/..." and this folder seems related to the sharing activity, I'll see whether or not this behavior has anything to do with caching.

Re: Script are changed by ESRI with g_ESRI_variable_1, g_ESRI_variable_2

simoxu — Mon, 13 Aug 2018 04:59:57 GMT

That makes sense.

I remember someone reported the similar issue when publishing a custom Geoprocessing to the server.

But my case is really about sharing a project with other team mates, all the tools are designed to run on the local machine rather than run as a web service. After they download it (or open it from portal/ArcGIS Online directly), the project folder structure are guaranteed and all the references to the datasource and Online resources should be ready to use.

Thanks

Re: Script are changed by ESRI with g_ESRI_variable_1, g_ESRI_variable_2

DanPatterson_Retired — Mon, 13 Aug 2018 10:13:52 GMT

from Package Project—Data Management toolbox | ArcGIS Desktop

did you remove the history?

And I would make sure that you have no reference to your folders as either a data source or a data destination since that will cause the tool to fail if people in your organization don't have access to, or permission to use, it

Re: Script are changed by ESRI with g_ESRI_variable_1, g_ESRI_variable_2

simoxu — Tue, 14 Aug 2018 03:04:45 GMT

Yes, I removed the history completely. I seriously think this is a bug in the Package Project tool.

It makes sense to analyze the code when you publish the geoprocessing tool as a service and make sure all the folders used in the code are accessible by the server, But as a project package, this process is not needed and the folders are relative to the project folder, which will be created when you unpack the project in ArcGIS Pro.

Re: Script are changed by ESRI with g_ESRI_variable_1, g_ESRI_variable_2

DanPatterson_Retired — Tue, 14 Aug 2018 03:11:40 GMT

I agree... it is that user folder with your name that still bugs me... don't know why it appears, unless your data were in it or it belongs to history

Re: Script are changed by ESRI with g_ESRI_variable_1, g_ESRI_variable_2

simoxu — Tue, 11 Sep 2018 06:42:19 GMT

A bug related to this topic has been confirmed by ESRI.

#BUG-000116650 Python script within the default toolbox in a project is appended with an invalid directory path after sharing the project as a project package.

Re: Script are changed by ESRI with g_ESRI_variable_1, g_ESRI_variable_2

DanMcCoy — Sat, 09 May 2020 00:26:08 GMT

simo xu‌, do you have any info about the similar issue when publishing a custom geoprocessing service?

Thanks,

Dan

Re: Script are changed by ESRI with g_ESRI_variable_1, g_ESRI_variable_2

DanMcCoy — Sat, 09 May 2020 00:30:50 GMT

I think I found it...
https://community.esri.com/thread/207534-why-does-arcgis-server-change-path-variable-to-gesrivariable1

Re: Script are changed by ESRI with g_ESRI_variable_1, g_ESRI_variable_2

myESRIUName — Sat, 09 Nov 2024 02:47:11 GMT

I know this thread is very old, but I thought I'd pass along some info related to it because I've just gone through a couple of days dealing with this too. I only just found the addition of a "g_ESRI_variable_1" variable to a script in a Project Package I had created because I was testing it to ensure what I had in mind would work. It didn't and I've spent two days trying to figure out why, but that's beside the point. I know now.

When I finally did stumble on this "g_ESRI_variable_1", I was reminded that I had run into the exact same problem about 10-15 years ago with ArcGIS Server 9.3 and again with 10.1. At both times, I was struggling with getting Geoprocessing Services to run properly after publishing. Eventually, looking into the v101 folder on the server, I discovered what ArcGIS Server was doing to the code during the publishing process each time. And each time, fixed the problems by "over-complicating" the things that the ArcGIS Server was changing erroneously. You'll get the idea in a moment. My point is that this is not a new problem only related to Pro. It's been in their source code for a while and they seem to feel it's a good thing to change code without informing the user. Otherwise, they would have done something about it by now.

What I've been struggling with over the last couple of days is that I have a project with scripts that I am putting in source control as a project package. The project's name is "my_project_name" and the first thing each script within it does when invoked is verify that it is running within the "my_project_name" project because that project is structurally set up as it needs to be for all of these scripts to work.

Unfortunately, ArcGIS Pro is finding this:

project_name = "my_project_name" aprx = arcpy.mp.ArcGISProject("CURRENT") pos = aprx.filePath.rfind(project_name) if pos < 0: arcpy.AddError("This script must be run within the '" + project_name + "' ArcGIS Pro project.") sys.exit()

The project_name variable is used for many things within the scripts and ArcGIS Pro is changing them all to use g_ESRI_variable_1 because the project_name variable is instantiated with a string that is the same project name that the ppks file explodes to:

g_ESRI_variable_1 = os.path.join(arcpy.env.scriptWorkspace,'..\\commondata\\my_project_name1')

Once I realized that was the problem, and recognized it was the same problem I'd beaten my head against the wall over (repeatedly) many years ago, I intro'd the same fix I did then. I "hid" the string in question by complicating the code so ESRI stopped trying to "fix" it for me. So, now I have:

# project_name = "my_project_name" # doesn't work project_name = "_".join(["my","project","name"])

Et voila.

Similarly, though not for this particular problem, but in previous ones, we got around some of the problems by providing our paths with "/" rather the "\" and using Python's os.path.normpath functionality to convert the code to Windows structure at run time. ArcGIS wouldn't change those or the ones when we built the paths using the os.path.join (or path.pathlib.join_path, now) but there had to be enough parameters passed that ArcGIS couldn't figure out the path. I'm guessing, then, that the reporter of this problem could probably try something like this (no testing, just off the top of my head) to break things up and confuse ArcGIS Pro enough that it wouldn't try to change the script:

# checkout_image_path= os.path.join(homefolder,"current_checkout_attachments")‍‍ checkout_image_path= os.path.normpath(homefolder) + r"\\current_checkout_attachments")‍‍

Anyway, just throwing this stuff out there in case someone else comes across this thread looking for ideas.

Good luck!!

jtm