Hi,
I’m hoping someone can shed some light on a problem I’ve been wrestling with. I have been unable get the Python 3 runtime for ArcGIS Server 10.7.1 on Linux running through a crontab job and wanted to see if anyone else has achieved what I am trying.
I have a bash shell script (cron_test.sh) running on Ubuntu 18.04.2 that runs the following commands:
export ARCGISHOME=/arcgis/server
. /home/arcgis/anaconda3/etc/profile.d/conda.sh
conda activate python36b
python /home/arcgis/arcgis-imagery/Geoplex/cron_test.py
and the cron_test.py script is only creating a file GDB:
import sys
import os
import arcpy
import datetime
import tempfile
import logging
TEMPDIR = tempfile.mkdtemp()
fgdb_name = "test.gdb"
fgdb_path = os.path.join(TEMPDIR, fgdb_name)
logging_level = logging.INFO
log_dir = '/tmp/logs'
now = datetime.datetime.strftime(datetime.datetime.now(), "%Y_%m_%d_%H_%M_%S")
logfile_name = "test" + '_' + now + '.log'
logging.basicConfig(filename=os.path.join(log_dir, logfile_name), level=logging_level)
for a in os.environ:
logging.info('Var: ' + str(a) + ' Value: ' + str(os.getenv(a)))
logging.info(f"\tCreating file geodatabase in {TEMPDIR}, called {fgdb_name}")
arcpy.CreateFileGDB_management(TEMPDIR, fgdb_name)
logging.info(f"\tCreated file geodatabase in {TEMPDIR}, called {fgdb_name}")
When I sudo into the arcgis user interactively and then run the shell script with the command:
Bash /path/to/cron_test.sh
The script runs and creates the file GDB fine:
INFO:root:Var: CONDA_SHLVL Value: 2
INFO:root:Var: LD_LIBRARY_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/lib:/arcgis/server/bin/wine/lib64:/arcgis/server/bin/wine/lib64/wine:/arcgis/server/bin/wine/lib64/wine/supp:/arcgis/server/framework/runtime/tomcat/bin:/arcgis/server/lib:
INFO:root:Var: LIBGL_DRIVERS_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/lib/dri
INFO:root:Var: LS_COLORS Value: rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
INFO:root:Var: CONDA_EXE Value: /home/arcgis/anaconda3/bin/conda
INFO:root:Var: FONTCONFIG_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/etc/fonts
INFO:root:Var: XVFBPATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb
INFO:root:Var: LESSCLOSE Value: /usr/bin/lesspipe %s %s
INFO:root:Var: SDEHOME Value: /arcgis/server
INFO:root:Var: LANG Value: C.UTF-8
INFO:root:Var: AGS_ORIG_PATH Value: /arcgis/server/bin:/home/arcgis/anaconda3/envs/python36b/bin:/home/arcgis/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
INFO:root:Var: SUDO_GID Value: 1000
INFO:root:Var: OLDPWD Value: /home/arcgis
INFO:root:Var: WINEPATH Value: /arcgis/server/bin/wine
INFO:root:Var: AGSSERVER Value: /arcgis/server/
INFO:root:Var: USERNAME Value: root
INFO:root:Var: CONDA_PREFIX Value: /home/arcgis/anaconda3/envs/python36b
INFO:root:Var: SUDO_COMMAND Value: /bin/su arcgis
INFO:root:Var: _CE_M Value:
INFO:root:Var: CLASSPATH Value: /arcgis/server/framework/lib/arcobjects.jar:
INFO:root:Var: USER Value: arcgis
INFO:root:Var: CONDA_PREFIX_1 Value: /home/arcgis/anaconda3
INFO:root:Var: arcgis_cloud_platform Value: aws
INFO:root:Var: PWD Value: /tmp/logs
INFO:root:Var: HOME Value: /home/arcgis
INFO:root:Var: CONDA_PYTHON_EXE Value: /home/arcgis/anaconda3/bin/python
INFO:root:Var: SUDO_USER Value: ubuntu
INFO:root:Var: ARCGISHOME Value: /arcgis/server
INFO:root:Var: _CE_CONDA Value:
INFO:root:Var: WINEDEBUG Value: -all
INFO:root:Var: TOMCATBIN Value: /arcgis/server/framework/runtime/tomcat/bin
INFO:root:Var: CONDA_PROMPT_MODIFIER Value: (python36b)
INFO:root:Var: SUDO_UID Value: 1000
INFO:root:Var: MAIL Value: /var/mail/arcgis
INFO:root:Var: ARCGIS_JAVA_HOME Value: /arcgis/server//framework/runtime/jre/
INFO:root:Var: TERM Value: xterm
INFO:root:Var: SHELL Value: /bin/bash
INFO:root:Var: FONTCONFIG_FILE Value: /arcgis/server/framework/runtime/xvfb/Xvfb/etc/fonts/fonts.conf
INFO:root:Var: SHLVL Value: 2
INFO:root:Var: AGS_ORIG_LD_LIBRARY_PATH Value: /arcgis/server/lib:
INFO:root:Var: LOGNAME Value: arcgis
INFO:root:Var: DOTWINE Value: /arcgis/server/framework/runtime/.wine
INFO:root:Var: PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/bin:/arcgis/server/bin/wine/bin:/arcgis/server//framework/runtime/jre//bin:/arcgis/server/framework/runtime/ArcGIS/Resources/ArcToolBox/Scripts:/arcgis/server/framework/runtime/ArcGIS/bin:/arcgis/server/bin:/home/arcgis/anaconda3/envs/python36b/bin:/home/arcgis/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
INFO:root:Var: CONDA_DEFAULT_ENV Value: python36b
INFO:root:Var: HOMEPATH Value: /home/arcgis
INFO:root:Var: ENGINEJAVA94 Value: /arcgis/server/
INFO:root:Var: WINEPREFIX Value: /arcgis/server/framework/runtime/.wine
INFO:root:Var: LESSOPEN Value: | /usr/bin/lesspipe %s
INFO:root:Var: _ Value: /home/arcgis/anaconda3/envs/python36b/bin/python
INFO:root: Creating file geodatabase in /tmp/tmp8dp6qygo, called test.gdb
INFO:root: Created file geodatabase in /tmp/tmp8dp6qygo, called test.gdb
The geodatabase is created fine and can be viewed. The environment variables are logged for the purpose of the next step.
I need to run an arcpy script periodically, so I have tried to call the bash script by adding it to crontab. This is where the problem starts.
I add the bash command to the arcgis user crontab file:
* * * * * bash /path/to/cron_test.sh
When the script runs it can import the arcpy library but when it gets to the CreateFileGDB command the script hangs and it doesn’t create the file GDB, no errors reported either.
INFO:root:Var: CONDA_SHLVL Value: 1
INFO:root:Var: LD_LIBRARY_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/lib:/arcgis/server/bin/wine/lib64:/arcgis/server/bin/wine/lib64/wine:/arcgis/server/bin/wine/lib64/wine/supp:/arcgis/server/framework/runtime/tomcat/bin:/arcgis/server/lib:
INFO:root:Var: LIBGL_DRIVERS_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/lib/dri
INFO:root:Var: CONDA_EXE Value: /home/arcgis/anaconda3/bin/conda
INFO:root:Var: FONTCONFIG_PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/etc/fonts
INFO:root:Var: XVFBPATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb
INFO:root:Var: SDEHOME Value: /arcgis/server
INFO:root:Var: LANG Value: C.UTF-8
INFO:root:Var: AGS_ORIG_PATH Value: /arcgis/server/bin:/home/arcgis/anaconda3/envs/python36b/bin:/home/arcgis/anaconda3/condabin:/usr/bin:/bin
INFO:root:Var: WINEPATH Value: /arcgis/server/bin/wine
INFO:root:Var: AGSSERVER Value: /arcgis/server/
INFO:root:Var: CONDA_PREFIX Value: /home/arcgis/anaconda3/envs/python36b
INFO:root:Var: _CE_M Value:
INFO:root:Var: CLASSPATH Value: /arcgis/server/framework/lib/arcobjects.jar:
INFO:root:Var: arcgis_cloud_platform Value: aws
INFO:root:Var: PWD Value: /home/arcgis
INFO:root:Var: HOME Value: /home/arcgis
INFO:root:Var: CONDA_PYTHON_EXE Value: /home/arcgis/anaconda3/bin/python
INFO:root:Var: ARCGISHOME Value: /arcgis/server
INFO:root:Var: _CE_CONDA Value:
INFO:root:Var: WINEDEBUG Value: -all
INFO:root:Var: TOMCATBIN Value: /arcgis/server/framework/runtime/tomcat/bin
INFO:root:Var: CONDA_PROMPT_MODIFIER Value: (python36b)
INFO:root:Var: ARCGIS_JAVA_HOME Value: /arcgis/server//framework/runtime/jre/
INFO:root:Var: SHELL Value: /bin/sh
INFO:root:Var: FONTCONFIG_FILE Value: /arcgis/server/framework/runtime/xvfb/Xvfb/etc/fonts/fonts.conf
INFO:root:Var: SHLVL Value: 1
INFO:root:Var: AGS_ORIG_LD_LIBRARY_PATH Value: /arcgis/server/lib:
INFO:root:Var: LOGNAME Value: arcgis
INFO:root:Var: DOTWINE Value: /arcgis/server/framework/runtime/.wine
INFO:root:Var: PATH Value: /arcgis/server/framework/runtime/xvfb/Xvfb/bin:/arcgis/server/bin/wine/bin:/arcgis/server//framework/runtime/jre//bin:/arcgis/server/framework/runtime/ArcGIS/Resources/ArcToolBox/Scripts:/arcgis/server/framework/runtime/ArcGIS/bin:/arcgis/server/bin:/home/arcgis/anaconda3/envs/python36b/bin:/home/arcgis/anaconda3/condabin:/usr/bin:/bin
INFO:root:Var: CONDA_DEFAULT_ENV Value: python36b
INFO:root:Var: HOMEPATH Value: /home/arcgis
INFO:root:Var: ENGINEJAVA94 Value: /arcgis/server/
INFO:root:Var: WINEPREFIX Value: /arcgis/server/framework/runtime/.wine
INFO:root:Var: _ Value: /home/arcgis/anaconda3/envs/python36b/bin/python
INFO:root: Creating file geodatabase in /tmp/tmpu2gxq8jy, called test.gdb
When crontab runs the command there are less variables coming through to the python script, but I have tried manually set them in the bash shell script with export commands so they are identical to the manual run and I still get the same result. This suggests the is more than setting environment variables that is preventing the arcpy commands from running.
Other things I have tried is running the shell commands directly in crontab job like:
SHELL=/bin/bash
BASH_ENV=/home/arcgis/anaconda3/etc/profile.d/conda.sh
* * * * * export ARCGISHOME=/arcgis/server; conda activate python36b; python /home/arcgis/arcgis-imagery/Geoplex/cron_test.py
I’ve tried loading the profile and bashrc files into the bash script:
. /home/arcgis/.profile
. /home/arcgis/.bashrc
I’ve taken out everything except the import arcpy and createFileGDB command in the python script, and also I’ve tried using the fish shell in place of bash, all to no avail.
With the behavior I’m getting can someone point me to how to troubleshoot this further? It would be greatly appreciated!
Anaconda version Anaconda3-2020.02-Linux-x86_64
Thanks,
Leon.
