Problems with downloads from Open Data site

[Anonymous User]

There appears to be a problem with our Open Geography portal (http://geoportal.statistics.gov.uk)

Symptoms

When downloading the shapefile format from the portal – there is nothing in the shapefile zip files – example –

http://geoportal.statistics.gov.uk/datasets/local-authority-districts-december-2016-full-clipped-bou... if you click the download the zip file is only 32.1KB.

This has happened to numerous other examples on the site. When I check the ArcGIS Server logs I get

SEVERE	Jun 21, 2018, 10:01:31 AM	<?xml version="1.0" encoding="UTF-8" ?><ExceptionReport xmlns="http://www.opengis.net/ows/1.1" ><Exception code="server.unknown" ><ExceptionText><![CDATA[java.net.MalformedURLException: no protocol: ]]></ExceptionText><ExceptionText><![CDATA[java.net.MalformedURLException: no protocol: ]]></ExceptionText></Exception></ExceptionReport>	Administrative_Boundaries/UK_Local_Authority_Districts_December_2016_Boundaries.MapServer
SEVERE	Jun 21, 2018, 10:01:31 AM	Failed to initialize ResponseStreamer object. no protocol:	Administrative_Boundaries/UK_Local_Authority_Districts_December_2016_Boundaries.MapServer

SEVERE

Jun 21, 2018, 10:01:31 AM

SOE exception during request processing 'com.esri.a4i.soe.api.SOEException: Feature Download Server backend is not available, please contact the system administrator. Most probably there are no feature types configured. Please proceed to service configuration page in ArcCatalog and select the feature types to be served.' com.esri.a4i.soe.api.SOEException: Feature Download Server backend is not available, please contact the system administrator. Most probably there are no feature types configured. Please proceed to service configuration page in ArcCatalog and select the feature types to be served. at com.esri.a4i.soe.common.xtraserver.client.XtraServerClient.queryXtraServer(XtraServerClient.java:92) at com.esri.a4i.soe.download.handler.WFSOperationHandler.queryXtraServer(WFSOperationHandler.java:17) at com.esri.a4i.soe.download.handler.WFSOperationHandler.execute(WFSOperationHandler.java:12) at com.esri.a4i.soe.download.handler.WFSRequestHandler.handleRequest(WFSRequestHandler.java:104) at com.esri.a4i.soe.download.handler.WFSRequestHandler.handleGet(WFSRequestHandler.java:67) at com.esri.a4i.soe.download.InternalDirectAccessRequest.handleRequestedResource(InternalDirectAccessRequest.java:59) at com.esri.a4i.soe.download.InternalDirectAccessRequest.processGetRequest(InternalDirectAccessRequest.java:46) at com.esri.a4i.soe.common.AbstractInternalSOERequest.process(AbstractInternalSOERequest.java:48) at com.esri.a4i.soe.common.AbstractInspireService.processRequest(AbstractInspireService.java:203) at com.esri.a4i.soe.common.AbstractInspireService.handleRESTRequest(AbstractInspireService.java:135) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.esri.arcgis.interop.NativeObjRef.nativeVtblInvokeNative(Native Method) at com.esri.arcgis.interop.NativeObjRef.nativeVtblInvoke(Unknown Source) at com.esri.arcgis.interop.NativeObjRef.invoke(Unknown Source) at com.esri.arcgis.interop.Dispatch.vtblInvoke(Unknown Source) at com.esri.arcgis.system.IRESTRequestHandlerProxy.handleRESTRequest(Unknown Source) at com.esri.arcgis.discovery.servicelib.impl.SOThread.a(SOThread.java:1062) at com.esri.arcgis.discovery.servicelib.impl.SOThread.a(SOThread.java:690) at com.esri.arcgis.discovery.servicelib.impl.SOThread.run(SOThread.java:446)

Administrative_Boundaries/UK_Local_Authority_Districts_December_2016_Boundaries.MapServer

SEVERE	Jun 21, 2018, 10:17:37 AM	Geodatabase error: The operation was attempted on an empty geometry.	Administrative_Boundaries/UK_Local_Authority_Districts_December_2016_Boundaries.MapServer
SEVERE	Jun 21, 2018, 10:17:37 AM	Geodatabase error: The operation was attempted on an empty geometry.	Administrative_Boundaries/UK_Local_Authority_Districts_December_2016_Boundaries.MapServer

Another example which I have just received is

I work with Bill Roberts at Swirrl and in our recent exploration of the data from the Open Geography Portal we came across a problem which we wanted to report.

The Travel To Work Areas (TTWA) listed under http://geoportal.statistics.gov.uk/datasets?q=ttwa%202011 all seem to have a problem with the shapefiles download. 

Once you download the shapefiles they are very small in size and seem to contain no boundary information. We have tried to process these shapefiles in R and load them into mapshaper.org but all unsuccessfully.

It would be great if you could take a look at this issue, or forward this to the team responsible for the portal.

When I check the boundaries at https://ons-inspire.esriuk.com/arcgis/ and pull the data direct into ArcMap they look fine and work so it doesn’t look like a problem with the INSPIRE ArcGIS Server – it looks like it could be the ArcGIS Hub (Open Data App) side that has the problem. The cache on the ArcGIS Hub side appears to have corrupted. Can you check with Esri Inc as a matter of urgency?

We have tried to reset the downloads in https://hub.arcgis.com/admin/data but with no effect. Our download caches are set to manual. This was done several months ago as the data never changes

Has anyone else reported problems with the other open data sites?

[DanielFenton1 · Accepted solution]

Thank you for bringing this to our attention. We have resolved the immediate issue causing this problem. It is unclear if the logs you are seeing from your server were related. We will be conducting more testing and analysis to better understand the root cause and prevent it from reoccurring. We apologize for the inconvenience.

Daniel Fenton

Software Engineer | ArcGIS Hub

[Anonymous User]

Thanks Daniel, 

[Natural_EnglandOpen_Data_Publi]

Hi Daniel,

I'm interested to hear what the fix was for this and why it occurred in the first place.

I experienced exactly the same behaviour with many of our ArcGIS Online hosted feature layers not downloading correctly in Shapefile format from the Natural England Open Data site https://naturalengland-defra.opendata.arcgis.com. This was around the same time (late June into early July) as experienced by Matt at ONS.  Many shapefiles were downloading without spatial objects, but not all hosted services were affected, which was unusual.

No change was made on the publisher side. Was this an issue experienced with the latest releases to ArcGIS Online and the ArcGIS Hub?

Has the KOOP toolset (which I believe you use to extract the features in multiple formats) been fully tested with the latest releases?

To test and find a resolution I worked through each hosted service (71 layers)  by downloading each and viewing them in ArcMap.  For the ones that downloaded as KB sized files with only an attribute table, I clicked 'Rebuild Indexes' in the Settings tab of each ArcGIS Online hosted item page, then clicked 'Reset Dataset and Downloads' against each related item on the ArcGIS Hub Data page.  Resetting downloads alone did not resolve the issue.  This resolved it but was a very laborious and time consuming task over two days and had an impact on my wellbeing.  This issue brought many of our public download services to a halt for approximately 1-2 weeks unbeknown to us.  Most of our end users still download shapefiles for translation to other formats/local analysis. We received negative customer feedback during this time.

We still have four affected layers, our largest most complex geometry datasets. I have a support request being worked on in the UK (Miguel Paredes) and in Redlands (Jarret Baker) to suggest next steps.  I have discussed with Miguel and he will add your name to the case for help.

Case no. #02137485 - Updating downloads in hub.

Andy Lee

Data Services Team

Natural England

[DanielFenton1]

Andy,

We recently reworked our support for native projection downloads. Shortly after releasing that feature, we received reports of downloads with empty geometries. We immediately rolled the feature back. However, some datasets became "stuck" in a bad state due to the way we store cached data to provide downloads without overwhelming your servers.

We worked to find and remove the corrupted caches, then applied a fix to the feature itself. That fix was adding an additional parameter to our GDAL command which does the heavy lifting with respect to producing downloadable files. We then re-released the new feature and have not had any additional reports of issues since that time.

Running the reindex operation for your entire data catalog does not actually reset any downloads. I'm a bit confused by: "Resetting downloads alone did not resolve the issue.  This resolved it but was a very laborious." Did clicking the reset button on each dataset resolve the issue or not? We do not include an option to rebuild all of your download caches at once as that would be an overwhelming amount of work for your servers to handle.

We are very sorry about the impact on you and your customers. Which parts of the process could we make more simple or automated? What information could we provide within the app to prevent you from doing your manual process?

As to the 4 affected layers, are they not downloading at all or are they downloading with empty geometry? Not downloading at all could be caused by any number of issues and it is best to continue on with support for a more specific diagnosis. Downloading with empty geometry is something the development team would want to look at right away.

Again, we apologize for the inconvenience caused for you and your customers. We work hard to provide a stable download system and we let you down here. We now have additional automated testing in place that will catch any regression of the specific error that occurred previously so it will not happen again.

Daniel Fenton

Software Engineer | ArcGIS Hub

[Natural_EnglandOpen_Data_Publi]

Thanks so much for the info above Daniel, and for sharing and admitting the issue with the corruption of services through the changes recently made to the platform.

I am aware you are in conversation with Miguel in the UK in support of this ongoing issue, and therefore have access to most of the details on the support enquiry.  Resetting the downloads in the Hub did not solve the issue, I had to click to rebuild the spatial indexes on the AGOL item settings pages too, as the datasets were downloading with no geometry.  Strange hey, as the services were behaving OK in the Map Viewer!

For the four remaining complex geometry layers with a large no. of vertices (derived from OS MasterMap Topography layer) I have now tried uploading them again with the default max record count of 1000. They are failing to download (stuck on Gathering Data). The details are on the support case. Miguel is helping with testing these. I will try reducing the max record counts to lower values next week to see if that helps. This helped resolve issues before with other services.  I suggest reducing to ≤500, what do you think?  Then I'll keep reducing the value until the downloads start to function....hopefully.  Each time I will click 'Reset Dataset and downloads' in the Hub Data page. For some data services I have very low max record count values , too low really in some cases to be honest.  The Hub does not function very well with datasets which include polygons with a large no. of vertices, especially derived from OS MasterMap Topography Layer.

Thanks for your continued help.

Andy