pfoppe@blm.gov_BLM_EGIS

ERROR 001461: Failed to package tiles (Can't load format DLL)

Discussion created by pfoppe@blm.gov_BLM_EGIS on Mar 26, 2018
Latest reply on Apr 5, 2018 by rkelley@blm.gov_BLM_EGIS

We have deployed an ArcGIS Server Mapping service (running 10.4.1 + security patches) and cached the service to LOD 14.  Our mobile users would like to use this service offline on their mobile devices (collector for arcgis), however we are having troubles with the exportTiles operation.  The server is returning an error that it failed to package tiles: 

Job Status je10e77c47e0b4c33b679f0fea1d60d96(Export Tiles)

Job ID: je10e77c47e0b4c33b679f0fea1d60d96_et

 

Job Status: esriJobFailed

 

Job Messages:
  • esriJobMessageTypeInformative: Submitted.
  • esriJobMessageTypeInformative: Executing...
  • esriJobMessageTypeInformative: Executing (Export Cache): ExportCache <FOLDER>/<SERVICE>:MapServer # COMPACT TRUE 1 TRUE # 1,2,3,4 FALSE "Feature Set" # {"requestMode":"REST","exportBy":"levelId","recompress":false,"recompressQuality":75}
  • esriJobMessageTypeInformative: Start Time: Mon Mar 26 17:33:00 2018
  • esriJobMessageTypeInformative: Finished:: 4 percent
  • esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
  • esriJobMessageTypeInformative: Finished:: 13 percent
  • esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
  • esriJobMessageTypeInformative: Finished:: 34 percent
  • esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
  • esriJobMessageTypeInformative: Finished:: 99 percent
  • esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
  • esriJobMessageTypeError: ERROR 001461: Failed to package tiles.
  • esriJobMessageTypeError: Can't load format DLL.
  • esriJobMessageTypeError: Failed to execute (Export Cache).
  • esriJobMessageTypeInformative: Failed at Mon Mar 26 17:33:04 2018 (Elapsed Time: 4.19 seconds)
  • esriJobMessageTypeError: Failed.

 

I was able to access the Esri tiled basemaps and export a tile package with the same input variables:

  • esriJobMessageTypeInformative: Executing (Export Cache): ExportCache World_Street_Map:MapServer # COMPACT TRUE 1 TRUE # 1,2,3,4 FALSE "Feature Set" # {"requestMode":"REST","exportBy":"levelId","recompress":false,"recompressQuality":90}
  • esriJobMessageTypeInformative: Start Time: Mon Mar 26 16:13:50 2018
  • esriJobMessageTypeInformative: Finished:: 2 percent
  • esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
  • esriJobMessageTypeInformative: Finished:: 8 percent
  • esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
  • esriJobMessageTypeInformative: Finished:: 28 percent
  • esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
  • esriJobMessageTypeInformative: Finished:: 99 percent
  • esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
  • esriJobMessageTypeInformative: Exported a package of 4220405 bytes.
  • esriJobMessageTypeInformative: Succeeded at Mon Mar 26 16:14:02 2018 (Elapsed Time: 11.46 seconds)

The only main differences I see between our two services is that the Esri online basemap tileInfo->format is JPEG while ours is MIXED.  Esri is also running 10.5.1 while we are on 10.4.1.  

 

I am able to export tiles if I disable the tile Package input variable (in blue below):

  • esriJobMessageTypeInformative: Executing (Export Cache): ExportCache <folder>/<service>:MapServer # COMPACT TRUE 1 FALSE # 1,2,3,4 FALSE "Feature Set" # {"requestMode":"REST","exportBy":"levelId","recompress":false,"recompressQuality":75}
  • esriJobMessageTypeInformative: Start Time: Mon Mar 26 17:22:09 2018
  • esriJobMessageTypeInformative: Finished:: 4 percent
  • esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
  • esriJobMessageTypeInformative: Finished:: 13 percent
  • esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
  • esriJobMessageTypeInformative: Finished:: 34 percent
  • esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
  • esriJobMessageTypeInformative: Finished:: 99 percent
  • esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
  • esriJobMessageTypeInformative: Succeeded at Mon Mar 26 17:22:13 2018 (Elapsed Time: 4.40 seconds)

I've reviewed the Export Tiles operation REST API doc - Export Tiles—ArcGIS REST API: Services Directory | ArcGIS for Developers 

 

and the Tool Reference error code documentation - 001461: Failed to package tiles.—Help | ArcGIS Desktop 

 

I've ensured that the target cache directory and output directory both have enough space and the headless ArcGIS Server AD service account has modify rights to both locations.  

 

ArcGIS Server logs did not provide much further visibility (although there was a few stack traces present in DEBUG mode): 

 

DEBUGMar 26, 2018, 5:44:39 PMorg.apache.jasper.JasperException: org.apache.jasper.JasperException: java.lang.NullPointerException at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:556) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:462) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:486) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:411) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:338) at com.esri.rf.RRequest.forward(RRequest.java:983) at com.esri.rf.RRequest.htmlForwardElseThrowHE(RRequest.java:552) at com.esri.client.app.templates.OperationTemplate.handleException(OperationTemplate.java:271) at com.esri.client.app.templates.OperationTemplate.execute(OperationTemplate.java:121) at com.esri.client.app.mapserver.MapExportTilesHandler.exportStatus(MapExportTilesHandler.java:399) at com.esri.client.app.mapserver.MapExportTilesHandler.service(MapExportTilesHandler.java:83) at com.esri.rf.RServlet.service(RServlet.java:116) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.jasper.JasperException: java.lang.NullPointerException at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:556) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:477) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:604) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:543) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:954) at org.apache.jsp.www.framework.common.template_jsp._jspService(template_jsp.java:156) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:439) ... 39 more Caused by: java.lang.NullPointerException at org.apache.jsp.www.client.gpserver.gpjob_jsp._jspService(gpjob_jsp.java:87) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:439) ... 52 moreRest
SEVEREMar 26, 2018, 5:44:39 PMUnable to process request. org.apache.jasper.JasperException: java.lang.NullPointerExceptionRest
DEBUGMar 26, 2018, 5:44:39 PMcom.esri.arcgis.discovery.json.JSONException: JSONObject["value"] not found. at com.esri.arcgis.discovery.json.JSONObject.get(JSONObject.java:369) at com.esri.arcgis.discovery.json.JSONObject.getString(JSONObject.java:496) at com.esri.client.app.mapserver.MapExportTilesHandler$5.doOperation(MapExportTilesHandler.java:373) at com.esri.client.app.templates.OperationTemplate.execute(OperationTemplate.java:119) at com.esri.client.app.mapserver.MapExportTilesHandler.exportStatus(MapExportTilesHandler.java:399) at com.esri.client.app.mapserver.MapExportTilesHandler.service(MapExportTilesHandler.java:83) at com.esri.rf.RServlet.service(RServlet.java:116) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745)Rest
SEVEREMar 26, 2018, 5:44:39 PMError getting status for job j1b0dc1ca4311489eb1f5fae8bd197455 on Export Cache JSONObject["value"] not found.Rest

 

I also reviewed windows event viewer on the 2 back end servers that are in the cluster hosting this service (and the caching service).  

 

We have many arcgis server environments (all running 10.4.1), and 1 is explicitly setup for cache builds (not for production use).  I tested this both on the original cache build server, and our public facing environment.  Both failed with the same results.  

 

Posting the community for any feedback/suggestions before I open an incident with Esri.  Thank you in advance!

 

Sean C Haile Huber, Daniel CStacey E CroweRyan Kelley

Outcomes