AnsweredAssumed Answered

Python 3 runtime ArcGIS Server Linux Crontab

Question asked by LeonAus on Jun 24, 2020
Latest reply on Jun 25, 2020 by bixb0012

Hi,

 

I’m hoping someone can shed some light on a problem I’ve been wrestling with. I have been unable get the Python 3 runtime for ArcGIS Server 10.7.1 on Linux running through a crontab job and wanted to see if anyone else has achieved what I am trying.

 

I have a bash shell script (cron_test.sh) running on Ubuntu 18.04.2 that runs the following commands:

 

export ARCGISHOME=/arcgis/server

. /home/arcgis/anaconda3/etc/profile.d/conda.sh

conda activate python36b

python /home/arcgis/arcgis-imagery/Geoplex/cron_test.py

 

and the cron_test.py script is only creating a file GDB:

 

            import sys

import os

import arcpy

import datetime

import tempfile

import logging

 

TEMPDIR = tempfile.mkdtemp()

 

fgdb_name = "test.gdb"

fgdb_path = os.path.join(TEMPDIR, fgdb_name)

 

logging_level = logging.INFO

log_dir = '/tmp/logs'

now = datetime.datetime.strftime(datetime.datetime.now(), "%Y_%m_%d_%H_%M_%S")

logfile_name = "test" + '_' + now + '.log'

logging.basicConfig(filename=os.path.join(log_dir, logfile_name), level=logging_level)

 

for a in os.environ:

    logging.info('Var: ' + str(a) + ' Value: ' + str(os.getenv(a)))

 

logging.info(f"\tCreating file geodatabase in {TEMPDIR}, called {fgdb_name}")

arcpy.CreateFileGDB_management(TEMPDIR, fgdb_name)

logging.info(f"\tCreated file geodatabase in {TEMPDIR}, called {fgdb_name}")

 

When I sudo into the arcgis user interactively and then run the shell script with the command:

 

Bash /path/to/cron_test.sh

 

The script runs and creates the file GDB fine:

           

            INFO:root:Var: CONDA_SHLVL Value: 2

INFO:root:Var: LD_LIBRARY_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/lib:/arcgis/server/bin/wine/lib64:/arcgis/server/bin/wine/lib64/wine:/arcgis/server/bin/wine/lib64/wine/supp:/arcgis/server/framework/runtime/tomcat/bin:/arcgis/server/lib:

INFO:root:Var: LIBGL_DRIVERS_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/lib/dri

INFO:root:Var: LS_COLORS Value: rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:

INFO:root:Var: CONDA_EXE Value: /home/arcgis/anaconda3/bin/conda

INFO:root:Var: FONTCONFIG_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/etc/fonts

INFO:root:Var: XVFBPATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb

INFO:root:Var: LESSCLOSE Value: /usr/bin/lesspipe %s %s

INFO:root:Var: SDEHOME Value: /arcgis/server

INFO:root:Var: LANG Value: C.UTF-8

INFO:root:Var: AGS_ORIG_PATH Value: /arcgis/server/bin:/home/arcgis/anaconda3/envs/python36b/bin:/home/arcgis/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

INFO:root:Var: SUDO_GID Value: 1000

INFO:root:Var: OLDPWD Value: /home/arcgis

INFO:root:Var: WINEPATH Value: /arcgis/server/bin/wine

INFO:root:Var: AGSSERVER Value: /arcgis/server/

INFO:root:Var: USERNAME Value: root

INFO:root:Var: CONDA_PREFIX Value: /home/arcgis/anaconda3/envs/python36b

INFO:root:Var: SUDO_COMMAND Value: /bin/su arcgis

INFO:root:Var: _CE_M Value:

INFO:root:Var: CLASSPATH Value: /arcgis/server/framework/lib/arcobjects.jar:

INFO:root:Var: USER Value: arcgis

INFO:root:Var: CONDA_PREFIX_1 Value: /home/arcgis/anaconda3

INFO:root:Var: arcgis_cloud_platform Value: aws

INFO:root:Var: PWD Value: /tmp/logs

INFO:root:Var: HOME Value: /home/arcgis

INFO:root:Var: CONDA_PYTHON_EXE Value: /home/arcgis/anaconda3/bin/python

INFO:root:Var: SUDO_USER Value: ubuntu

INFO:root:Var: ARCGISHOME Value: /arcgis/server

INFO:root:Var: _CE_CONDA Value:

INFO:root:Var: WINEDEBUG Value: -all

INFO:root:Var: TOMCATBIN Value: /arcgis/server/framework/runtime/tomcat/bin

INFO:root:Var: CONDA_PROMPT_MODIFIER Value: (python36b)

INFO:root:Var: SUDO_UID Value: 1000

INFO:root:Var: MAIL Value: /var/mail/arcgis

INFO:root:Var: ARCGIS_JAVA_HOME Value: /arcgis/server//framework/runtime/jre/

INFO:root:Var: TERM Value: xterm

INFO:root:Var: SHELL Value: /bin/bash

INFO:root:Var: FONTCONFIG_FILE Value: /arcgis/server/framework/runtime/xvfb/Xvfb/etc/fonts/fonts.conf

INFO:root:Var: SHLVL Value: 2

INFO:root:Var: AGS_ORIG_LD_LIBRARY_PATH Value: /arcgis/server/lib:

INFO:root:Var: LOGNAME Value: arcgis

INFO:root:Var: DOTWINE Value: /arcgis/server/framework/runtime/.wine

INFO:root:Var: PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/bin:/arcgis/server/bin/wine/bin:/arcgis/server//framework/runtime/jre//bin:/arcgis/server/framework/runtime/ArcGIS/Resources/ArcToolBox/Scripts:/arcgis/server/framework/runtime/ArcGIS/bin:/arcgis/server/bin:/home/arcgis/anaconda3/envs/python36b/bin:/home/arcgis/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

INFO:root:Var: CONDA_DEFAULT_ENV Value: python36b

INFO:root:Var: HOMEPATH Value: /home/arcgis

INFO:root:Var: ENGINEJAVA94 Value: /arcgis/server/

INFO:root:Var: WINEPREFIX Value: /arcgis/server/framework/runtime/.wine

INFO:root:Var: LESSOPEN Value: | /usr/bin/lesspipe %s

INFO:root:Var: _ Value: /home/arcgis/anaconda3/envs/python36b/bin/python

INFO:root:      Creating file geodatabase in /tmp/tmp8dp6qygo, called test.gdb

INFO:root:      Created file geodatabase in /tmp/tmp8dp6qygo, called test.gdb

 

The geodatabase is created fine and can be viewed. The environment variables are logged for the purpose of the next step.

 

I need to run an arcpy script periodically, so I have tried to call the bash script by adding it to crontab. This is where the problem starts.

 

I add the bash command to the arcgis user crontab file:

 

              * * * * * bash /path/to/cron_test.sh

 

When the script runs it can import the arcpy library but when it gets to the CreateFileGDB command the script hangs and it doesn’t create the file GDB, no errors reported either.

 

            INFO:root:Var: CONDA_SHLVL Value: 1

INFO:root:Var: LD_LIBRARY_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/lib:/arcgis/server/bin/wine/lib64:/arcgis/server/bin/wine/lib64/wine:/arcgis/server/bin/wine/lib64/wine/supp:/arcgis/server/framework/runtime/tomcat/bin:/arcgis/server/lib:

INFO:root:Var: LIBGL_DRIVERS_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/lib/dri

INFO:root:Var: CONDA_EXE Value: /home/arcgis/anaconda3/bin/conda

INFO:root:Var: FONTCONFIG_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/etc/fonts

INFO:root:Var: XVFBPATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb

INFO:root:Var: SDEHOME Value: /arcgis/server

INFO:root:Var: LANG Value: C.UTF-8

INFO:root:Var: AGS_ORIG_PATH Value: /arcgis/server/bin:/home/arcgis/anaconda3/envs/python36b/bin:/home/arcgis/anaconda3/condabin:/usr/bin:/bin

INFO:root:Var: WINEPATH Value: /arcgis/server/bin/wine

INFO:root:Var: AGSSERVER Value: /arcgis/server/

INFO:root:Var: CONDA_PREFIX Value: /home/arcgis/anaconda3/envs/python36b

INFO:root:Var: _CE_M Value:

INFO:root:Var: CLASSPATH Value: /arcgis/server/framework/lib/arcobjects.jar:

INFO:root:Var: arcgis_cloud_platform Value: aws

INFO:root:Var: PWD Value: /home/arcgis

INFO:root:Var: HOME Value: /home/arcgis

INFO:root:Var: CONDA_PYTHON_EXE Value: /home/arcgis/anaconda3/bin/python

INFO:root:Var: ARCGISHOME Value: /arcgis/server

INFO:root:Var: _CE_CONDA Value:

INFO:root:Var: WINEDEBUG Value: -all

INFO:root:Var: TOMCATBIN Value: /arcgis/server/framework/runtime/tomcat/bin

INFO:root:Var: CONDA_PROMPT_MODIFIER Value: (python36b)

INFO:root:Var: ARCGIS_JAVA_HOME Value: /arcgis/server//framework/runtime/jre/

INFO:root:Var: SHELL Value: /bin/sh

INFO:root:Var: FONTCONFIG_FILE Value: /arcgis/server/framework/runtime/xvfb/Xvfb/etc/fonts/fonts.conf

INFO:root:Var: SHLVL Value: 1

INFO:root:Var: AGS_ORIG_LD_LIBRARY_PATH Value: /arcgis/server/lib:

INFO:root:Var: LOGNAME Value: arcgis

INFO:root:Var: DOTWINE Value: /arcgis/server/framework/runtime/.wine

INFO:root:Var: PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/bin:/arcgis/server/bin/wine/bin:/arcgis/server//framework/runtime/jre//bin:/arcgis/server/framework/runtime/ArcGIS/Resources/ArcToolBox/Scripts:/arcgis/server/framework/runtime/ArcGIS/bin:/arcgis/server/bin:/home/arcgis/anaconda3/envs/python36b/bin:/home/arcgis/anaconda3/condabin:/usr/bin:/bin

INFO:root:Var: CONDA_DEFAULT_ENV Value: python36b

INFO:root:Var: HOMEPATH Value: /home/arcgis

INFO:root:Var: ENGINEJAVA94 Value: /arcgis/server/

INFO:root:Var: WINEPREFIX Value: /arcgis/server/framework/runtime/.wine

INFO:root:Var: _ Value: /home/arcgis/anaconda3/envs/python36b/bin/python

INFO:root:      Creating file geodatabase in /tmp/tmpu2gxq8jy, called test.gdb

 

When crontab runs the command there are less variables coming through to the python script, but I have tried manually set them in the bash shell script with export commands so they are identical to the manual run and I still get the same result. This suggests the is more than setting environment variables that is preventing the arcpy commands from running.

 

Other things I have tried is running the shell commands directly in crontab job like:

SHELL=/bin/bash

BASH_ENV=/home/arcgis/anaconda3/etc/profile.d/conda.sh

* * * * * export ARCGISHOME=/arcgis/server; conda activate python36b; python /home/arcgis/arcgis-imagery/Geoplex/cron_test.py

 

I’ve tried loading the profile and bashrc files into the bash script:

 

              . /home/arcgis/.profile

              . /home/arcgis/.bashrc

 

I’ve taken out everything except the import arcpy and createFileGDB command in the python script, and also I’ve tried using the fish shell in place of bash, all to no avail.

 

With the behavior I’m getting can someone point me to how to troubleshoot this further? It would be greatly appreciated! 

 

Anaconda version Anaconda3-2020.02-Linux-x86_64

 

Thanks,

Leon.

Outcomes