topic Re: ERROR 001461: Failed to package tiles (Can't load format DLL) in ArcGIS Enterprise Questions

ERROR 001461: Failed to package tiles (Can't load format DLL)

pfoppe — Mon, 26 Mar 2018 23:58:10 GMT

We have deployed an ArcGIS Server Mapping service (running 10.4.1 + security patches) and cached the service to LOD 14. Our mobile users would like to use this service offline on their mobile devices (collector for arcgis), however we are having troubles with the exportTiles operation. The server is returning an error that it failed to package tiles:

Job Status je10e77c47e0b4c33b679f0fea1d60d96(Export Tiles)
Job ID: je10e77c47e0b4c33b679f0fea1d60d96_et
Job Status: esriJobFailed
Job Messages:
esriJobMessageTypeInformative: Submitted.
esriJobMessageTypeInformative: Executing...
esriJobMessageTypeInformative: Executing (Export Cache): ExportCache <FOLDER>/<SERVICE>:MapServer # COMPACT TRUE 1 TRUE # 1,2,3,4 FALSE "Feature Set" # {"requestMode":"REST","exportBy":"levelId","recompress":false,"recompressQuality":75}
esriJobMessageTypeInformative: Start Time: Mon Mar 26 17:33:00 2018
esriJobMessageTypeInformative: Finished:: 4 percent
esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
esriJobMessageTypeInformative: Finished:: 13 percent
esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
esriJobMessageTypeInformative: Finished:: 34 percent
esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
esriJobMessageTypeInformative: Finished:: 99 percent
esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
esriJobMessageTypeError: ERROR 001461: Failed to package tiles.
esriJobMessageTypeError: Can't load format DLL.
esriJobMessageTypeError: Failed to execute (Export Cache).
esriJobMessageTypeInformative: Failed at Mon Mar 26 17:33:04 2018 (Elapsed Time: 4.19 seconds)
esriJobMessageTypeError: Failed.

I was able to access the Esri tiled basemaps and export a tile package with the same input variables:

esriJobMessageTypeInformative: Executing (Export Cache): ExportCache World_Street_Map:MapServer # COMPACT TRUE 1 TRUE # 1,2,3,4 FALSE "Feature Set" # {"requestMode":"REST","exportBy":"levelId","recompress":false,"recompressQuality":90}
esriJobMessageTypeInformative: Start Time: Mon Mar 26 16:13:50 2018
esriJobMessageTypeInformative: Finished:: 2 percent
esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
esriJobMessageTypeInformative: Finished:: 8 percent
esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
esriJobMessageTypeInformative: Finished:: 28 percent
esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
esriJobMessageTypeInformative: Finished:: 99 percent
esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
esriJobMessageTypeInformative: Exported a package of 4220405 bytes.
esriJobMessageTypeInformative: Succeeded at Mon Mar 26 16:14:02 2018 (Elapsed Time: 11.46 seconds)

The only main differences I see between our two services is that the Esri online basemap tileInfo->format is JPEG while ours is MIXED. Esri is also running 10.5.1 while we are on 10.4.1.

I am able to export tiles if I disable the tile Package input variable (in blue below):

esriJobMessageTypeInformative: Executing (Export Cache): ExportCache <folder>/<service>:MapServer # COMPACT TRUE 1 FALSE # 1,2,3,4 FALSE "Feature Set" # {"requestMode":"REST","exportBy":"levelId","recompress":false,"recompressQuality":75}
esriJobMessageTypeInformative: Start Time: Mon Mar 26 17:22:09 2018
esriJobMessageTypeInformative: Finished:: 4 percent
esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
esriJobMessageTypeInformative: Finished:: 13 percent
esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
esriJobMessageTypeInformative: Finished:: 34 percent
esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
esriJobMessageTypeInformative: Finished:: 99 percent
esriJobMessageTypeInformative: Estimated Time Remaining: Calculating...
esriJobMessageTypeInformative: Succeeded at Mon Mar 26 17:22:13 2018 (Elapsed Time: 4.40 seconds)

I've reviewed the Export Tiles operation REST API doc - Export Tiles—ArcGIS REST API: Services Directory | ArcGIS for Developers

and the Tool Reference error code documentation - 001461: Failed to package tiles.—Help | ArcGIS Desktop

I've ensured that the target cache directory and output directory both have enough space and the headless ArcGIS Server AD service account has modify rights to both locations.

ArcGIS Server logs did not provide much further visibility (although there was a few stack traces present in DEBUG mode):

DEBUG	Mar 26, 2018, 5:44:39 PM	org.apache.jasper.JasperException: org.apache.jasper.JasperException: java.lang.NullPointerException at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:556) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:462) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:486) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:411) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:338) at com.esri.rf.RRequest.forward(RRequest.java:983) at com.esri.rf.RRequest.htmlForwardElseThrowHE(RRequest.java:552) at com.esri.client.app.templates.OperationTemplate.handleException(OperationTemplate.java:271) at com.esri.client.app.templates.OperationTemplate.execute(OperationTemplate.java:121) at com.esri.client.app.mapserver.MapExportTilesHandler.exportStatus(MapExportTilesHandler.java:399) at com.esri.client.app.mapserver.MapExportTilesHandler.service(MapExportTilesHandler.java:83) at com.esri.rf.RServlet.service(RServlet.java:116) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.jasper.JasperException: java.lang.NullPointerException at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:556) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:477) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:604) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:543) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:954) at org.apache.jsp.www.framework.common.template_jsp._jspService(template_jsp.java:156) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:439) ... 39 more Caused by: java.lang.NullPointerException at org.apache.jsp.www.client.gpserver.gpjob_jsp._jspService(gpjob_jsp.java:87) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:439) ... 52 more	Rest
SEVERE	Mar 26, 2018, 5:44:39 PM	Unable to process request. org.apache.jasper.JasperException: java.lang.NullPointerException	Rest
DEBUG	Mar 26, 2018, 5:44:39 PM	com.esri.arcgis.discovery.json.JSONException: JSONObject["value"] not found. at com.esri.arcgis.discovery.json.JSONObject.get(JSONObject.java:369) at com.esri.arcgis.discovery.json.JSONObject.getString(JSONObject.java:496) at com.esri.client.app.mapserver.MapExportTilesHandler$5.doOperation(MapExportTilesHandler.java:373) at com.esri.client.app.templates.OperationTemplate.execute(OperationTemplate.java:119) at com.esri.client.app.mapserver.MapExportTilesHandler.exportStatus(MapExportTilesHandler.java:399) at com.esri.client.app.mapserver.MapExportTilesHandler.service(MapExportTilesHandler.java:83) at com.esri.rf.RServlet.service(RServlet.java:116) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745)	Rest
SEVERE	Mar 26, 2018, 5:44:39 PM	Error getting status for job j1b0dc1ca4311489eb1f5fae8bd197455 on Export Cache JSONObject["value"] not found.	Rest

I also reviewed windows event viewer on the 2 back end servers that are in the cluster hosting this service (and the caching service).

We have many arcgis server environments (all running 10.4.1), and 1 is explicitly setup for cache builds (not for production use). I tested this both on the original cache build server, and our public facing environment. Both failed with the same results.

Posting the community for any feedback/suggestions before I open an incident with Esri. Thank you in advance!

Sean C Haile‌ Huber, Daniel C‌Stacey E Crowe‌Ryan Kelley‌

Re: ERROR 001461: Failed to package tiles (Can't load format DLL)

Anonymous User — Tue, 27 Mar 2018 11:15:42 GMT

I don't have a whole lot of insight here, as I don't ever use the ExportTiles option from a cached map service, at least not directly. However, I ran it for Oregon's internal basemap (https://web.gismaps.or.blm.doi.net/arcgis/rest/services/Basemaps/Cached_ORWA_BLM_Carto_Basemap/MapServer/exportTiles) and it it was successful (url good for today) for Level 10 tiles.

Is there another process you can have your users run or you can run for them to package the TPKs? There are other options.(ie. arcpy.ExportMapServerCache_server). To get TPKs from Oregon's internal base map, I just run a script I wrote to chunk out the TPKs by resource area, and have had no issues. I can send it your way if that would help.

What is the service you are trying to export tiles from that is failing?

Re: ERROR 001461: Failed to package tiles (Can't load format DLL)

pfoppe — Thu, 05 Apr 2018 17:37:53 GMT

Thanks for the response Ryan. The ExportTiles is called from the mobile devices when attempting to take this 'basemap' offline. The service we have this enabled on is our external test environment at - lands/BLM_Natl_SMA_Cached_without_PriUnk (MapServer)

Re: ERROR 001461: Failed to package tiles (Can't load format DLL)

pfoppe — Thu, 05 Apr 2018 19:23:38 GMT

Update to this thread. After a painful trial and error process, we tracked down the issue with a large Map Name length (that is derived from the MXD Data Frame value which defaults to "Layers"). This service in particular has a quite long name at 96 characters currently - "BLM National Surface Management Agency - without PriUnk - National Geospatial Data Asset (NGDA)"

It appears that the name of the .tpk file that is generated is based on this value. The name also has hyphens and parenthesis which was another possible issue, however we did test with a shorter name with both hyphens and parenthesis with success.

Here is an example .tpk file path that is staged in the output location (this one worked because the name was short enough): \\<server>\<share>$\ArcGISServer\directories\arcgisoutput\lands\BLM_Natl_SMA_Cached_without_PriUnk_paran_and_hyphen_MapServer\_agsLMapCacheLF440\test - test (test).tpk

Here is an example that failed: \\<server>\<share>$\ArcGISServer\directories\arcgisoutput\lands\BLM_Natl_SMA_Cached_without_PriUnk_ph_target250_MapServer\_agsLMapCacheLEA9\target250aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa - test (test)\_alllayers\L03

We will consider shrinking this value down on our production environment.

BOTTOM LINE: Consider impacts to business workflows (including mobile) when changing the data frame value in services that are published.

Re: ERROR 001461: Failed to package tiles (Can't load format DLL)

Anonymous User — Thu, 05 Apr 2018 19:26:35 GMT

I just ran the export manually, via the ExportTiles command, and was successful at Level 9. Unsuccessful at Level 10. The export tiles threshold looks to be set @ 100,000, and Level 10 tried exporting 103k+, but level 9 was less.

So, not sure if the extent is factoring in here, if the tiles allowed need to be increased?? I don't get the mobile piece and what it actually does, so am no help here on that front.

Success:

Job Status j5b6317cc6a6c4301a92b69c88a0c8bfe(Export Tiles)

Failure:

Job Status j91598c6d8a58431b9da5ae6785c0faa2(Export Tiles)

Re: ERROR 001461: Failed to package tiles (Can't load format DLL)

Anonymous User — Thu, 05 Apr 2018 19:26:55 GMT

Just saw this;)