Python 3 runtime ArcGIS Server Linux Crontab

[LeonAus]

Hi,

I’m hoping someone can shed some light on a problem I’ve been wrestling with. I have been unable get the Python 3 runtime for ArcGIS Server 10.7.1 on Linux running through a crontab job and wanted to see if anyone else has achieved what I am trying.

I have a bash shell script (cron_test.sh) running on Ubuntu 18.04.2 that runs the following commands:

export ARCGISHOME=/arcgis/server

. /home/arcgis/anaconda3/etc/profile.d/conda.sh

conda activate python36b

python /home/arcgis/arcgis-imagery/Geoplex/cron_test.py

and the cron_test.py script is only creating a file GDB:

            import sys

import os

import arcpy

import datetime

import tempfile

import logging

TEMPDIR = tempfile.mkdtemp()

fgdb_name = "test.gdb"

fgdb_path = os.path.join(TEMPDIR, fgdb_name)

logging_level = logging.INFO

log_dir = '/tmp/logs'

now = datetime.datetime.strftime(datetime.datetime.now(), "%Y_%m_%d_%H_%M_%S")

logfile_name = "test" + '_' + now + '.log'

logging.basicConfig(filename=os.path.join(log_dir, logfile_name), level=logging_level)

for a in os.environ:

    logging.info('Var: ' + str(a) + ' Value: ' + str(os.getenv(a)))

logging.info(f"\tCreating file geodatabase in {TEMPDIR}, called {fgdb_name}")

arcpy.CreateFileGDB_management(TEMPDIR, fgdb_name)

logging.info(f"\tCreated file geodatabase in {TEMPDIR}, called {fgdb_name}")

When I sudo into the arcgis user interactively and then run the shell script with the command:

Bash /path/to/cron_test.sh

The script runs and creates the file GDB fine:

            INFO:root:Var: CONDA_SHLVL Value: 2

INFO:root:Var: LD_LIBRARY_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/lib:/arcgis/server/bin/wine/lib64:/arcgis/server/bin/wine/lib64/wine:/arcgis/server/bin/wine/lib64/wine/supp:/arcgis/server/framework/runtime/tomcat/bin:/arcgis/server/lib:

INFO:root:Var: LIBGL_DRIVERS_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/lib/dri

INFO:root:Var: LS_COLORS Value: rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:

INFO:root:Var: CONDA_EXE Value: /home/arcgis/anaconda3/bin/conda

INFO:root:Var: FONTCONFIG_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/etc/fonts

INFO:root:Var: XVFBPATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb

INFO:root:Var: LESSCLOSE Value: /usr/bin/lesspipe %s %s

INFO:root:Var: SDEHOME Value: /arcgis/server

INFO:root:Var: LANG Value: C.UTF-8

INFO:root:Var: AGS_ORIG_PATH Value: /arcgis/server/bin:/home/arcgis/anaconda3/envs/python36b/bin:/home/arcgis/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

INFO:root:Var: SUDO_GID Value: 1000

INFO:root:Var: OLDPWD Value: /home/arcgis

INFO:root:Var: WINEPATH Value: /arcgis/server/bin/wine

INFO:root:Var: AGSSERVER Value: /arcgis/server/

INFO:root:Var: USERNAME Value: root

INFO:root:Var: CONDA_PREFIX Value: /home/arcgis/anaconda3/envs/python36b

INFO:root:Var: SUDO_COMMAND Value: /bin/su arcgis

INFO:root:Var: _CE_M Value:

INFO:root:Var: CLASSPATH Value: /arcgis/server/framework/lib/arcobjects.jar:

INFO:root:Var: USER Value: arcgis

INFO:root:Var: CONDA_PREFIX_1 Value: /home/arcgis/anaconda3

INFO:root:Var: arcgis_cloud_platform Value: aws

INFO:root:Var: PWD Value: /tmp/logs

INFO:root:Var: HOME Value: /home/arcgis

INFO:root:Var: CONDA_PYTHON_EXE Value: /home/arcgis/anaconda3/bin/python

INFO:root:Var: SUDO_USER Value: ubuntu

INFO:root:Var: ARCGISHOME Value: /arcgis/server

INFO:root:Var: _CE_CONDA Value:

INFO:root:Var: WINEDEBUG Value: -all

INFO:root:Var: TOMCATBIN Value: /arcgis/server/framework/runtime/tomcat/bin

INFO:root:Var: CONDA_PROMPT_MODIFIER Value: (python36b)

INFO:root:Var: SUDO_UID Value: 1000

INFO:root:Var: MAIL Value: /var/mail/arcgis

INFO:root:Var: ARCGIS_JAVA_HOME Value: /arcgis/server//framework/runtime/jre/

INFO:root:Var: TERM Value: xterm

INFO:root:Var: SHELL Value: /bin/bash

INFO:root:Var: FONTCONFIG_FILE Value: /arcgis/server/framework/runtime/xvfb/Xvfb/etc/fonts/fonts.conf

INFO:root:Var: SHLVL Value: 2

INFO:root:Var: AGS_ORIG_LD_LIBRARY_PATH Value: /arcgis/server/lib:

INFO:root:Var: LOGNAME Value: arcgis

INFO:root:Var: DOTWINE Value: /arcgis/server/framework/runtime/.wine

INFO:root:Var: PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/bin:/arcgis/server/bin/wine/bin:/arcgis/server//framework/runtime/jre//bin:/arcgis/server/framework/runtime/ArcGIS/Resources/ArcToolBox/Scripts:/arcgis/server/framework/runtime/ArcGIS/bin:/arcgis/server/bin:/home/arcgis/anaconda3/envs/python36b/bin:/home/arcgis/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

INFO:root:Var: CONDA_DEFAULT_ENV Value: python36b

INFO:root:Var: HOMEPATH Value: /home/arcgis

INFO:root:Var: ENGINEJAVA94 Value: /arcgis/server/

INFO:root:Var: WINEPREFIX Value: /arcgis/server/framework/runtime/.wine

INFO:root:Var: LESSOPEN Value: | /usr/bin/lesspipe %s

INFO:root:Var: _ Value: /home/arcgis/anaconda3/envs/python36b/bin/python

INFO:root:      Creating file geodatabase in /tmp/tmp8dp6qygo, called test.gdb

INFO:root:      Created file geodatabase in /tmp/tmp8dp6qygo, called test.gdb

The geodatabase is created fine and can be viewed. The environment variables are logged for the purpose of the next step.

I need to run an arcpy script periodically, so I have tried to call the bash script by adding it to crontab. This is where the problem starts.

I add the bash command to the arcgis user crontab file:

              * * * * * bash /path/to/cron_test.sh

When the script runs it can import the arcpy library but when it gets to the CreateFileGDB command the script hangs and it doesn’t create the file GDB, no errors reported either.

            INFO:root:Var: CONDA_SHLVL Value: 1

INFO:root:Var: LD_LIBRARY_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/lib:/arcgis/server/bin/wine/lib64:/arcgis/server/bin/wine/lib64/wine:/arcgis/server/bin/wine/lib64/wine/supp:/arcgis/server/framework/runtime/tomcat/bin:/arcgis/server/lib:

INFO:root:Var: LIBGL_DRIVERS_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/lib/dri

INFO:root:Var: CONDA_EXE Value: /home/arcgis/anaconda3/bin/conda

INFO:root:Var: FONTCONFIG_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/etc/fonts

INFO:root:Var: XVFBPATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb

INFO:root:Var: SDEHOME Value: /arcgis/server

INFO:root:Var: LANG Value: C.UTF-8

INFO:root:Var: AGS_ORIG_PATH Value: /arcgis/server/bin:/home/arcgis/anaconda3/envs/python36b/bin:/home/arcgis/anaconda3/condabin:/usr/bin:/bin

INFO:root:Var: WINEPATH Value: /arcgis/server/bin/wine

INFO:root:Var: AGSSERVER Value: /arcgis/server/

INFO:root:Var: CONDA_PREFIX Value: /home/arcgis/anaconda3/envs/python36b

INFO:root:Var: _CE_M Value:

INFO:root:Var: CLASSPATH Value: /arcgis/server/framework/lib/arcobjects.jar:

INFO:root:Var: arcgis_cloud_platform Value: aws

INFO:root:Var: PWD Value: /home/arcgis

INFO:root:Var: HOME Value: /home/arcgis

INFO:root:Var: CONDA_PYTHON_EXE Value: /home/arcgis/anaconda3/bin/python

INFO:root:Var: ARCGISHOME Value: /arcgis/server

INFO:root:Var: _CE_CONDA Value:

INFO:root:Var: WINEDEBUG Value: -all

INFO:root:Var: TOMCATBIN Value: /arcgis/server/framework/runtime/tomcat/bin

INFO:root:Var: CONDA_PROMPT_MODIFIER Value: (python36b)

INFO:root:Var: ARCGIS_JAVA_HOME Value: /arcgis/server//framework/runtime/jre/

INFO:root:Var: SHELL Value: /bin/sh

INFO:root:Var: FONTCONFIG_FILE Value: /arcgis/server/framework/runtime/xvfb/Xvfb/etc/fonts/fonts.conf

INFO:root:Var: SHLVL Value: 1

INFO:root:Var: AGS_ORIG_LD_LIBRARY_PATH Value: /arcgis/server/lib:

INFO:root:Var: LOGNAME Value: arcgis

INFO:root:Var: DOTWINE Value: /arcgis/server/framework/runtime/.wine

INFO:root:Var: PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/bin:/arcgis/server/bin/wine/bin:/arcgis/server//framework/runtime/jre//bin:/arcgis/server/framework/runtime/ArcGIS/Resources/ArcToolBox/Scripts:/arcgis/server/framework/runtime/ArcGIS/bin:/arcgis/server/bin:/home/arcgis/anaconda3/envs/python36b/bin:/home/arcgis/anaconda3/condabin:/usr/bin:/bin

INFO:root:Var: CONDA_DEFAULT_ENV Value: python36b

INFO:root:Var: HOMEPATH Value: /home/arcgis

INFO:root:Var: ENGINEJAVA94 Value: /arcgis/server/

INFO:root:Var: WINEPREFIX Value: /arcgis/server/framework/runtime/.wine

INFO:root:Var: _ Value: /home/arcgis/anaconda3/envs/python36b/bin/python

INFO:root:      Creating file geodatabase in /tmp/tmpu2gxq8jy, called test.gdb

When crontab runs the command there are less variables coming through to the python script, but I have tried manually set them in the bash shell script with export commands so they are identical to the manual run and I still get the same result. This suggests the is more than setting environment variables that is preventing the arcpy commands from running.

Other things I have tried is running the shell commands directly in crontab job like:

SHELL=/bin/bash

BASH_ENV=/home/arcgis/anaconda3/etc/profile.d/conda.sh

* * * * * export ARCGISHOME=/arcgis/server; conda activate python36b; python /home/arcgis/arcgis-imagery/Geoplex/cron_test.py

I’ve tried loading the profile and bashrc files into the bash script:

              . /home/arcgis/.profile

              . /home/arcgis/.bashrc

I’ve taken out everything except the import arcpy and createFileGDB command in the python script, and also I’ve tried using the fish shell in place of bash, all to no avail.

With the behavior I’m getting can someone point me to how to troubleshoot this further? It would be greatly appreciated! 

Anaconda version Anaconda3-2020.02-Linux-x86_64

Thanks,

Leon.

[JoshuaBixby]

You have done a lot of great troubleshooting work, and have narrowed down the issue enough that I would reach out to Esri Support.  Although there is a chance a GeoNet user might have some good suggestions or feedback for you, you will probably get better feedback at this point by going through Support.

[SiCasley]

Hi Leon - did you ever get any support from Esri on this? I have also run into exactly the same issue.

By sending the standard and error output from my cron job to a text file:

   * * * * * conda activate /home/arcgis/envs; python /home/arcgis/my_test.py >> /home/arcgis/cron_test.txt 2>&1

I was able to get this additional error message which seems to happen when the first call to arcpy is made:

   Fatal Python error: Py_Initialize: can't initialize sys standard streams
   OSError: [WinError 6] Invalid handle

   Current thread 0x0000024c (most recent call first):

[SiCasley]

We're working with Esri (Canada) support on this... possibly an issue with Wine. Still no resolution for 10.7.1, but looks like the issue is fixed with 10.8.1 Python 3 runtime. 

[JohnEhlen]

I'm experiencing this exact issue on v10.9.1. Did you ever find a resolution? 

[Jay_Gregory]

Did you ever figure out what was happening? Running into this same issue at 10.9.1 as well. Running script manually works fine but thought crontab receiving the same errors listed by @SiCasley 

[Jay_Gregory]

With respect to this specific issue, I found that is occurs whenever you pipe output of a Python script with an arcpy command.  The script will execute fine until it reaches the arcpy command (import arcpy will work though).  

So python myarcpyscript.py will execute fine, but python myarcpyscript.py >> myarcpylog.log will terminate ungracefully with Fatal Python error: Py_Initialize: can't initialize sys standard streams
   OSError: [WinError 6] Invalid handle. Esri support confirmed this is an issue at 10.9.1 (and maybe earlier), but was fixed at 11.0.  

[LeonAus]

Simon, thanks for following up on this. I am back on the project that I ran into this problem with and I can confirm with 10.8.1 I am able to create a new file geodatabase through a crontab job.

Unfortunately there are a few ArcGIS objects that my script needs to create and I had hoped that the fix for the geodatabase would have fixed all creating commands but I'm unable to create a raster mosaic inside the gdb, or create a cloud storage connection file. Looks like I'll be going back to handing off those tasks to a geoprocessing service.