We have deployed an ArcGIS Server Mapping service (running 10.4.1 + security patches) and cached the service to LOD 14. Our mobile users would like to use this service offline on their mobile devices (collector for arcgis), however we are having troubles with the exportTiles operation. The server is returning an error that it failed to package tiles:
Job Status je10e77c47e0b4c33b679f0fea1d60d96(Export Tiles)
Job ID: je10e77c47e0b4c33b679f0fea1d60d96_et
Job Status: esriJobFailed
Job Messages:- esriJobMessageTypeInformative: Submitted.
- esriJobMessageTypeInformative: Executing...
- esriJobMessageTypeInformative: Executing (Export Cache): ExportCache <FOLDER>/<SERVICE>:MapServer # COMPACT TRUE 1 TRUE # 1,2,3,4 FALSE "Feature Set" # {"requestMode":"REST","exportBy":"levelId","recompress":false,"recompressQuality":75}
- esriJobMessageTypeInformative: Start Time: Mon Mar 26 17:33:00 2018
- esriJobMessageTypeInformative: Finished:: 4 percent
- esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
- esriJobMessageTypeInformative: Finished:: 13 percent
- esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
- esriJobMessageTypeInformative: Finished:: 34 percent
- esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
- esriJobMessageTypeInformative: Finished:: 99 percent
- esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
- esriJobMessageTypeError: ERROR 001461: Failed to package tiles.
- esriJobMessageTypeError: Can't load format DLL.
- esriJobMessageTypeError: Failed to execute (Export Cache).
- esriJobMessageTypeInformative: Failed at Mon Mar 26 17:33:04 2018 (Elapsed Time: 4.19 seconds)
- esriJobMessageTypeError: Failed.
I was able to access the Esri tiled basemaps and export a tile package with the same input variables:
- esriJobMessageTypeInformative: Executing (Export Cache): ExportCache World_Street_Map:MapServer # COMPACT TRUE 1 TRUE # 1,2,3,4 FALSE "Feature Set" # {"requestMode":"REST","exportBy":"levelId","recompress":false,"recompressQuality":90}
- esriJobMessageTypeInformative: Start Time: Mon Mar 26 16:13:50 2018
- esriJobMessageTypeInformative: Finished:: 2 percent
- esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
- esriJobMessageTypeInformative: Finished:: 8 percent
- esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
- esriJobMessageTypeInformative: Finished:: 28 percent
- esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
- esriJobMessageTypeInformative: Finished:: 99 percent
- esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
- esriJobMessageTypeInformative: Exported a package of 4220405 bytes.
- esriJobMessageTypeInformative: Succeeded at Mon Mar 26 16:14:02 2018 (Elapsed Time: 11.46 seconds)
The only main differences I see between our two services is that the Esri online basemap tileInfo->format is JPEG while ours is MIXED. Esri is also running 10.5.1 while we are on 10.4.1.
I am able to export tiles if I disable the tile Package input variable (in blue below):
- esriJobMessageTypeInformative: Executing (Export Cache): ExportCache <folder>/<service>:MapServer # COMPACT TRUE 1 FALSE # 1,2,3,4 FALSE "Feature Set" # {"requestMode":"REST","exportBy":"levelId","recompress":false,"recompressQuality":75}
- esriJobMessageTypeInformative: Start Time: Mon Mar 26 17:22:09 2018
- esriJobMessageTypeInformative: Finished:: 4 percent
- esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
- esriJobMessageTypeInformative: Finished:: 13 percent
- esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
- esriJobMessageTypeInformative: Finished:: 34 percent
- esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
- esriJobMessageTypeInformative: Finished:: 99 percent
- esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
- esriJobMessageTypeInformative: Succeeded at Mon Mar 26 17:22:13 2018 (Elapsed Time: 4.40 seconds)
I've reviewed the Export Tiles operation REST API doc - Export Tiles—ArcGIS REST API: Services Directory | ArcGIS for Developers
and the Tool Reference error code documentation - 001461: Failed to package tiles.—Help | ArcGIS Desktop
I've ensured that the target cache directory and output directory both have enough space and the headless ArcGIS Server AD service account has modify rights to both locations.
ArcGIS Server logs did not provide much further visibility (although there was a few stack traces present in DEBUG mode):
| DEBUG | Mar 26, 2018, 5:44:39 PM | org.apache.jasper.JasperException: org.apache.jasper.JasperException: java.lang.NullPointerException at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:556) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:462) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:486) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:411) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:338) at com.esri.rf.RRequest.forward(RRequest.java:983) at com.esri.rf.RRequest.htmlForwardElseThrowHE(RRequest.java:552) at com.esri.client.app.templates.OperationTemplate.handleException(OperationTemplate.java:271) at com.esri.client.app.templates.OperationTemplate.execute(OperationTemplate.java:121) at com.esri.client.app.mapserver.MapExportTilesHandler.exportStatus(MapExportTilesHandler.java:399) at com.esri.client.app.mapserver.MapExportTilesHandler.service(MapExportTilesHandler.java:83) at com.esri.rf.RServlet.service(RServlet.java:116) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.jasper.JasperException: java.lang.NullPointerException at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:556) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:477) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:604) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:543) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:954) at org.apache.jsp.www.framework.common.template_jsp._jspService(template_jsp.java:156) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:439) ... 39 more Caused by: java.lang.NullPointerException at org.apache.jsp.www.client.gpserver.gpjob_jsp._jspService(gpjob_jsp.java:87) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:439) ... 52 more | Rest |
| SEVERE | Mar 26, 2018, 5:44:39 PM | Unable to process request. org.apache.jasper.JasperException: java.lang.NullPointerException | Rest |
| DEBUG | Mar 26, 2018, 5:44:39 PM | com.esri.arcgis.discovery.json.JSONException: JSONObject["value"] not found. at com.esri.arcgis.discovery.json.JSONObject.get(JSONObject.java:369) at com.esri.arcgis.discovery.json.JSONObject.getString(JSONObject.java:496) at com.esri.client.app.mapserver.MapExportTilesHandler$5.doOperation(MapExportTilesHandler.java:373) at com.esri.client.app.templates.OperationTemplate.execute(OperationTemplate.java:119) at com.esri.client.app.mapserver.MapExportTilesHandler.exportStatus(MapExportTilesHandler.java:399) at com.esri.client.app.mapserver.MapExportTilesHandler.service(MapExportTilesHandler.java:83) at com.esri.rf.RServlet.service(RServlet.java:116) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) | Rest |
| SEVERE | Mar 26, 2018, 5:44:39 PM | Error getting status for job j1b0dc1ca4311489eb1f5fae8bd197455 on Export Cache JSONObject["value"] not found. | Rest |
I also reviewed windows event viewer on the 2 back end servers that are in the cluster hosting this service (and the caching service).
We have many arcgis server environments (all running 10.4.1), and 1 is explicitly setup for cache builds (not for production use). I tested this both on the original cache build server, and our public facing environment. Both failed with the same results.
Posting the community for any feedback/suggestions before I open an incident with Esri. Thank you in advance!
Sean C Haile Huber, Daniel CStacey E CroweRyan Kelley