Latest Contributions by pfoppe

Our organization is actively testing an ArcGIS Enterprise solution and we plan to build many federated servers to host 'user managed data' (think file geodatabases or Enterprise geodatabases that are usually part of some sort of data standard and have a fairly defined data lifecycle).  The federated servers may be setup based on our organization units (think a server for each major 'office' or 'program') and we plan to setup the federated servers with Restricted Publishing to prevent staff from 1 office (or customer base) to publish in another offices environment - reference: Administer a federated server—Portal for ArcGIS (10.6) | ArcGIS Enterprise  In order for this to work the portal role privilege "publish server based layers" (portal:publisher:publishServerServices) has to be enabled for the end users to publish to the federated servers.  Each of the federated servers can be configured with 'restricted publishing' to only allow users in a specific portal group to publish to that environment, however it seems that a member who is in a portal role that has the publish server-based layers privilege granted can also publish server based layers to the hosting server.  A couple of our staff members (including myself) have inadvertently published to the hosting server and as a result, the memory utilization was quite high causing cascading performance impacts.     Hence the Idea: Provide a setting to "Restrict Publishing Server-Based Layers" to an ArcGIS Enterprise Hosting Server   Basically want to restrict any 'server based layers' and designate the hosting server exclusively for the hosted services.   Thanks for the consideration. Reference - Can you setup a "Hosting Server" and "Restricted Publishing"?  ... View more

Our organization is actively testing an ArcGIS Enterprise solution and we plan to build many federated servers to host 'user managed data' (think file geodatabases or Enterprise geodatabases that are usually part of some sort of data standard and have a fairly defined data lifecycle).  The federated servers may be setup based on our organization units (think a server for each major 'office' or 'program') and we plan to setup the federated servers with Restricted Publishing to prevent staff from 1 office to publish in another offices environment - reference: Administer a federated server—Portal for ArcGIS (10.6) | ArcGIS Enterprise  This is all fine so far, and allows us to restrict our publishers so they only have access to the server we have setup for them, however it seems that a member who is in a portal role that has the publish server-based layers privilege granted can publish to the hosting server.  A couple of our staff members (including myself) have inadvertently published to the hosting server and as a result, the memory utilization was quite high causing cascading performance impacts.   Hence the question: Is there a way to setup the hosting server with 'restricted publishing' like you can a federated server?  Basically want to restrict any 'federated services' and designate this server exclusively for the hosted services.   Thanks! ... View more

Our organization has built a few python based windows services to complete some automated tasks; these windows services are installed on the same server running ArcGIS Server so they can use the underlying python deployment and tie into arcpy.  These windows services have functioned along side ArcGIS Server quite well until the v10.5 release.  At the 10.5 release, we are starting to experience issues.  Primarily that... if the python based windows service is started before the ArcGIS Server service, then all the ArcSOC.exe's fail on launch.  We can see the ArcSOC.exe briefly show up taskmgr but they disappear and throw the following exception in the ArcGIS Server manager logs -  Level Time Message Source Machine Code Time Elapsed User Name Process Thread Method Name DEBUG Nov 26, 2018, 5:25:38 PM Cleaning up previous service instance and server context. System/PublishingTools.GPServer <SERVER>.<DOMAIN> 9999 3512 78 DEBUG Nov 26, 2018, 5:25:38 PM com.esri.arcgis.discovery.servicelib.AGSException: activation failed; nested exception is: java.rmi.activation.ActivationException: object activation failed after 2 tries; nested exception is: java.rmi.ConnectException: Connection refused to host: <XX.XXX.XX.XX>; nested exception is: java.net.ConnectException: Connection refused: connect at com.esri.arcgis.discovery.servicelib.ServerContextFactory.createContext(ServerContextFactory.java:198) at com.esri.arcgis.discovery.ejb.util.EJBBase.initService(EJBBase.java:425) at com.esri.arcgis.discovery.ejb.util.EJBBase.init(EJBBase.java:277) at com.esri.arcgis.discovery.ejb.impl.GPServerSyncBean.init(GPServerSyncBean.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162) at org.apache.openejb.core.interceptor.ReflectionInvocationContext$LifecycleInvocation.invoke(ReflectionInvocationContext.java:194) at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144) at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:164) at org.apache.openejb.monitoring.StatsInterceptor.PostConstruct(StatsInterceptor.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162) at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144) at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:122) at org.apache.openejb.core.stateless.StatelessInstanceManager.ceateInstance(StatelessInstanceManager.java:237) at org.apache.openejb.core.stateless.StatelessInstanceManager.createInstance(StatelessInstanceManager.java:448) at org.apache.openejb.core.stateless.StatelessInstanceManager.access$100(StatelessInstanceManager.java:72) at org.apache.openejb.core.stateless.StatelessInstanceManager$BackgroundThread.run(StatelessInstanceManager.java:507) Caused by: java.rmi.activation.ActivateFailedException: activation failed; nested exception is: java.rmi.activation.ActivationException: object activation failed after 2 tries; nested exception is: java.rmi.ConnectException: Connection refused to host: <XX.XXX.XX.XX>; nested exception is: java.net.ConnectException: Connection refused: connect at sun.rmi.server.ActivatableRef.activate(ActivatableRef.java:304) at sun.rmi.server.ActivatableRef.invoke(ActivatableRef.java:133) at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227) at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179) at com.sun.proxy.$Proxy42.setProductCode(Unknown Source) at com.esri.arcgis.discovery.servicelib.ServerContextFactory.a(ServerContextFactory.java:255) at com.esri.arcgis.discovery.servicelib.ServerContextFactory.createContext(ServerContextFactory.java:195) ... 23 more Caused by: java.rmi.activation.ActivationException: object activation failed after 2 tries; nested exception is: java.rmi.ConnectException: Connection refused to host: <XX.XXX.XX.XX>; nested exception is: java.net.ConnectException: Connection refused: connect at sun.rmi.server.Activation$GroupEntry.activate(Activation.java:1184) at sun.rmi.server.Activation$ActivatorImpl.activate(Activation.java:406) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346) at sun.rmi.transport.Transport$1.run(Transport.java:200) at sun.rmi.transport.Transport$1.run(Transport.java:197) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:196) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162) at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227) at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179) at com.sun.proxy.$Proxy41.activate(Unknown Source) at java.rmi.activation.ActivationID.activate(ActivationID.java:118) at sun.rmi.server.ActivatableRef.activate(ActivatableRef.java:277) ... 29 more Caused by: java.rmi.ConnectException: Connection refused to host: <XX.XXX.XX.XX>; nested exception is: java.net.ConnectException: Connection refused: connect at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619) at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130) at java.rmi.activation.ActivationGroup_Stub.newInstance(Unknown Source) at sun.rmi.server.Activation$ObjectEntry.activate(Activation.java:1455) at sun.rmi.server.Activation$GroupEntry.activate(Activation.java:1141) at sun.rmi.server.Activation$ActivatorImpl.activate(Activation.java:406) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346) at sun.rmi.transport.Transport$1.run(Transport.java:200) at sun.rmi.transport.Transport$1.run(Transport.java:197) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:196) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused: connect at java.net.DualStackPlainSocketImpl.connect0(Native Method) at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at java.net.Socket.<init>(Socket.java:434) at java.net.Socket.<init>(Socket.java:211) at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148) at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613) ... 23 more System/PublishingToolsEx.GPServer <SERVER>.<DOMAIN> 9999 3512 83 DEBUG Nov 26, 2018, 5:25:38 PM Could not initialize service 'System.PublishingToolsEx.GPServer'. activation failed; nested exception is: java.rmi.activation.ActivationException: object activation failed after 2 tries; nested exception is: java.rmi.ConnectException: Connection refused to host: <XX.XXX.XX.XX>; nested exception is: java.net.ConnectException: Connection refused: connect System/PublishingToolsEx.GPServer <SERVER>.<DOMAIN> 8275 3512 83 SEVERE Nov 26, 2018, 5:25:38 PM Could not initialize service 'System.PublishingToolsEx.GPServer'. System/PublishingToolsEx.GPServer <SERVER>.<DOMAIN> 8275 3512 83 Here is a quick shell/example of a python based windows service that we have built:  logging_level = "DEBUG" logfile = r'C:\tmp\A-arcpy-pysvc-test.log' check_interval = 30 #standard imports import sys import os import logging from logging import handlers import traceback import socket import win32serviceutil import win32service import win32event import servicemanager exception_throw=False try:     import arcpy except:     sys.path.append(r'C:\Program Files\ArcGIS\Server\arcpy')     sys.path.append(r'C:\Program Files\ArcGIS\Server\bin')     sys.path.append(r'C:\Program Files\ArcGIS\Server\ArcToolbox\Scripts')     import arcpy     exception_throw = True #################################### ###     Classes/Functions     ### #################################### class WinNtSvc (win32serviceutil.ServiceFramework):     _svc_name_ = "A-arcpy-pysvc-test"     _svc_display_name_ = "A Python ArcPY Service Test"     def __init__(self,args):         win32serviceutil.ServiceFramework.__init__(self,args)         self.hWaitStop = win32event.CreateEvent(None,0,0,None)     def SvcStop(self):         self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)         win32event.SetEvent(self.hWaitStop)         logging.info("***** STOPPING WINDOWS NT SERVICE ON HOST '%s' *****"%socket.gethostname())     def SvcDoRun(self):         servicemanager.LogMsg(             servicemanager.EVENTLOG_INFORMATION_TYPE,                 servicemanager.PYS_SERVICE_STARTED,                 (self._svc_name_,'')         )         self.main()     def main(self):         ############################         ###     SETUP         ###         ############################         ###    SETUP LOGGING ###         logging_levels={'DEBUG':logging.DEBUG,                         'INFO':logging.INFO,                         'WARNING':logging.WARNING,                         'ERROR':logging.ERROR,                         'CRITICAL':logging.CRITICAL                         }         lu_log_level=logging_levels[logging_level] if logging_levels.has_key(logging_level) else "WARNING"         print "logging to %s"%logfile         # logging.basicConfig(format='[%(asctime)s][%(levelname)s] %(message)s', datefmt='%Y/%m/%d %H:%M:%S', level=lu_log_level, filename=logfile, filemode="a")         # 1 MB = 1048576 bytes         logger = logging.getLogger()         logger.setLevel(lu_log_level)         filehandler = logging.handlers.RotatingFileHandler(logfile)         formatter=logging.Formatter('[%(asctime)s][%(levelname)s][%(name)s][%(lineno)d] %(message)s', datefmt='%Y/%m/%d %H:%M:%S')         filehandler.setFormatter(formatter)         # write to file handler and standard out         logging.getLogger().addHandler(filehandler)         logging.getLogger().addHandler(logging.StreamHandler())         logging.info("***** STARTING '{}' LOOP ON HOST '{}' *****".format(self._svc_name_,socket.gethostname()))         if not logging_levels.has_key(logging_level):             logging.warning("Failed to determine logging level.  Defaulted to 'WARNING'  Options are: %s"%",".join(logging_levels.keys()))         ############################         ###     PRE-LOOP CODE     ###         ############################         ############################         ###     LOOPING ROUTINE     ###         ############################         logging.info("arcpy loaded from {}".format(arcpy))         if exception_throw:             logging.warning("unable to load arcpy without adjusting sys.path to: ")             for p in sys.path:                 logging.warning("  - {}".format(p))         logging.info("starting check every %s seconds"%check_interval)         rc=None         while rc != win32event.WAIT_OBJECT_0:             logging.info("Loop")             logging.info("Next check in %s seconds"%check_interval)             rc = win32event.WaitForSingleObject(self.hWaitStop, check_interval*1000) ##provides an interface to register the service. You can register the service with the system by running #python <service>.py install ##one line of executable code if __name__ == '__main__':     win32serviceutil.HandleCommandLine(WinNtSvc) ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ To install that service, save it as a file (ex: win_svc_sample.py), install the 64-bit version of pywin32 for python 2.7 and run this command from the windows command line as administrator:  python.exe win_svc_sample.py install‍‍‍ We've tried a variety of things to resolve including:  Installing a virtual environment with the pywin32 modules and creating the service from the virtualenv Installing ArcGIS Desktop along side server Tested on both 10.4.1, 10.5.1, and 10.6.1 on windows 2008 and 2012.   We can run a windows service just fine as long as we do not import arcpy.  If we import arcpy then the ArcSOC.exe fail to launch.  We were able to achieve success by installing ArcGIS Desktop along side the server, and can confirm that if arcpy is loaded from <module 'arcpy' from 'C:\Program Files (x86)\ArcGIS\Desktop10.6\ArcPy\arcpy\__init__.pyc'> then it will function when co-located on the same server as ArcGIS Server.  However, if desktop is not there, then arcpy is loaded from <module 'arcpy' from '<C:\Program Files\ArcGIS\Server\ArcPy\arcpy\__init__.pyc>'then the ArcSOC.exe's fail.   Thoughts from the community?  Is anyone else running python based windows services on the same server as ArcGIS for Server?  Any idea why importing arcpy from a windows service would prevent arcgis server from functioning? We can move these windows services to a new host running ArcGIS Desktop with background geoprocessing in the short term, however we have larger plans to co-locate with server at multiple locations in the future and would really like to ensure compatibility with server.  The error java.rmi.activation.ActivationException: object activation failed after 2 tries; nested exception is: java.rmi.ConnectException: Connection refused to host: <XX.XXX.XX.XX>; nested exception is: java.net.ConnectException: Connection refused: connect Suggests a firewall block (all are disabled and this works without the python based windows service) or a different component is failing which is listening for an active connection.   Interesting enough.. if the server is started first, then the python windows service, everything appears to work as expected.  Including stopping/re-starting arcgis server services (the ArcSOC.exe do not fail to launch as long as the server is running correctly).  Thanks for any help/advise.   ... View more

We have deployed an ArcGIS Server Mapping service (running 10.4.1 + security patches) and cached the service to LOD 14.  Our mobile users would like to use this service offline on their mobile devices (collector for arcgis), however we are having troubles with the exportTiles operation.  The server is returning an error that it failed to package tiles:  Job Status je10e77c47e0b4c33b679f0fea1d60d96(Export Tiles) Job ID: je10e77c47e0b4c33b679f0fea1d60d96_et Job Status: esriJobFailed Job Messages: esriJobMessageTypeInformative: Submitted. esriJobMessageTypeInformative: Executing... esriJobMessageTypeInformative: Executing (Export Cache): ExportCache <FOLDER>/<SERVICE>:MapServer # COMPACT TRUE 1 TRUE # 1,2,3,4 FALSE "Feature Set" # {"requestMode":"REST","exportBy":"levelId","recompress":false,"recompressQuality":75} esriJobMessageTypeInformative: Start Time: Mon Mar 26 17:33:00 2018 esriJobMessageTypeInformative: Finished:: 4 percent esriJobMessageTypeInformative: Estimated Time Remaining: Calculating... esriJobMessageTypeInformative: Finished:: 13 percent esriJobMessageTypeInformative: Estimated Time Remaining: Calculating... esriJobMessageTypeInformative: Finished:: 34 percent esriJobMessageTypeInformative: Estimated Time Remaining: Calculating... esriJobMessageTypeInformative: Finished:: 99 percent esriJobMessageTypeInformative: Estimated Time Remaining: Calculating... esriJobMessageTypeError: ERROR 001461: Failed to package tiles. esriJobMessageTypeError: Can't load format DLL. esriJobMessageTypeError: Failed to execute (Export Cache). esriJobMessageTypeInformative: Failed at Mon Mar 26 17:33:04 2018 (Elapsed Time: 4.19 seconds) esriJobMessageTypeError: Failed. I was able to access the Esri tiled basemaps and export a tile package with the same input variables: esriJobMessageTypeInformative: Executing (Export Cache): ExportCache World_Street_Map:MapServer # COMPACT TRUE 1 TRUE # 1,2,3,4 FALSE "Feature Set" # {"requestMode":"REST","exportBy":"levelId","recompress":false,"recompressQuality":90} esriJobMessageTypeInformative: Start Time: Mon Mar 26 16:13:50 2018 esriJobMessageTypeInformative: Finished:: 2 percent esriJobMessageTypeInformative: Estimated Time Remaining: Calculating... esriJobMessageTypeInformative: Finished:: 8 percent esriJobMessageTypeInformative: Estimated Time Remaining: Calculating... esriJobMessageTypeInformative: Finished:: 28 percent esriJobMessageTypeInformative: Estimated Time Remaining: Calculating... esriJobMessageTypeInformative: Finished:: 99 percent esriJobMessageTypeInformative: Estimated Time Remaining: Calculating... esriJobMessageTypeInformative: Exported a package of 4220405 bytes. esriJobMessageTypeInformative: Succeeded at Mon Mar 26 16:14:02 2018 (Elapsed Time: 11.46 seconds) The only main differences I see between our two services is that the Esri online basemap tileInfo->format is JPEG while ours is MIXED.  Esri is also running 10.5.1 while we are on 10.4.1.   I am able to export tiles if I disable the tile Package input variable (in blue below): esriJobMessageTypeInformative: Executing (Export Cache): ExportCache <folder>/<service>:MapServer # COMPACT TRUE 1 FALSE # 1,2,3,4 FALSE "Feature Set" # {"requestMode":"REST","exportBy":"levelId","recompress":false,"recompressQuality":75} esriJobMessageTypeInformative: Start Time: Mon Mar 26 17:22:09 2018 esriJobMessageTypeInformative: Finished:: 4 percent esriJobMessageTypeInformative: Estimated Time Remaining: Calculating... esriJobMessageTypeInformative: Finished:: 13 percent esriJobMessageTypeInformative: Estimated Time Remaining: Calculating... esriJobMessageTypeInformative: Finished:: 34 percent esriJobMessageTypeInformative: Estimated Time Remaining: Calculating... esriJobMessageTypeInformative: Finished:: 99 percent esriJobMessageTypeInformative: Estimated Time Remaining: Calculating... esriJobMessageTypeInformative: Succeeded at Mon Mar 26 17:22:13 2018 (Elapsed Time: 4.40 seconds) I've reviewed the Export Tiles operation REST API doc - Export Tiles—ArcGIS REST API: Services Directory | ArcGIS for Developers  and the Tool Reference error code documentation - 001461: Failed to package tiles.—Help | ArcGIS Desktop  I've ensured that the target cache directory and output directory both have enough space and the headless ArcGIS Server AD service account has modify rights to both locations.   ArcGIS Server logs did not provide much further visibility (although there was a few stack traces present in DEBUG mode):  DEBUG Mar 26, 2018, 5:44:39 PM org.apache.jasper.JasperException: org.apache.jasper.JasperException: java.lang.NullPointerException at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:556) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:462) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:486) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:411) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:338) at com.esri.rf.RRequest.forward(RRequest.java:983) at com.esri.rf.RRequest.htmlForwardElseThrowHE(RRequest.java:552) at com.esri.client.app.templates.OperationTemplate.handleException(OperationTemplate.java:271) at com.esri.client.app.templates.OperationTemplate.execute(OperationTemplate.java:121) at com.esri.client.app.mapserver.MapExportTilesHandler.exportStatus(MapExportTilesHandler.java:399) at com.esri.client.app.mapserver.MapExportTilesHandler.service(MapExportTilesHandler.java:83) at com.esri.rf.RServlet.service(RServlet.java:116) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.jasper.JasperException: java.lang.NullPointerException at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:556) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:477) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:604) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:543) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:954) at org.apache.jsp.www.framework.common.template_jsp._jspService(template_jsp.java:156) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:439) ... 39 more Caused by: java.lang.NullPointerException at org.apache.jsp.www.client.gpserver.gpjob_jsp._jspService(gpjob_jsp.java:87) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:439) ... 52 more Rest SEVERE Mar 26, 2018, 5:44:39 PM Unable to process request. org.apache.jasper.JasperException: java.lang.NullPointerException Rest DEBUG Mar 26, 2018, 5:44:39 PM com.esri.arcgis.discovery.json.JSONException: JSONObject["value"] not found. at com.esri.arcgis.discovery.json.JSONObject.get(JSONObject.java:369) at com.esri.arcgis.discovery.json.JSONObject.getString(JSONObject.java:496) at com.esri.client.app.mapserver.MapExportTilesHandler$5.doOperation(MapExportTilesHandler.java:373) at com.esri.client.app.templates.OperationTemplate.execute(OperationTemplate.java:119) at com.esri.client.app.mapserver.MapExportTilesHandler.exportStatus(MapExportTilesHandler.java:399) at com.esri.client.app.mapserver.MapExportTilesHandler.service(MapExportTilesHandler.java:83) at com.esri.rf.RServlet.service(RServlet.java:116) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Rest SEVERE Mar 26, 2018, 5:44:39 PM Error getting status for job j1b0dc1ca4311489eb1f5fae8bd197455 on Export Cache JSONObject["value"] not found. Rest I also reviewed windows event viewer on the 2 back end servers that are in the cluster hosting this service (and the caching service).   We have many arcgis server environments (all running 10.4.1), and 1 is explicitly setup for cache builds (not for production use).  I tested this both on the original cache build server, and our public facing environment.  Both failed with the same results.   Posting the community for any feedback/suggestions before I open an incident with Esri.  Thank you in advance! Sean C Haile‌ Huber, Daniel C‌Stacey E Crowe‌Ryan Kelley‌ ... View more