topic Re: AGO Open Data download issues in ArcGIS Hub Questions

AGO Open Data download issues

ChristopherBoland — Tue, 16 Dec 2014 19:32:19 GMT

Since the new update this morning, i've been testing my develpment site out and I've come up with a few issues:

When trying to download one dataset "Something went wrong with the server. Please try again or report this issue to the team." (Dataset | FWC GIS & Mapping Data Downloads )
While trying to download another large dataset, which I know has been a problem in the past, I only get 1000 of the 2328 records returned in my downloaded dataset (Dataset | FWC GIS & Mapping Data Downloads). I think this is because I tried to download this dataset before the most recent update and maybe the zipped shapefile was cached on ESRI's server with the max number of features returned by the server, but there's no instruction on how to clear that cached zip file.
When downloading a zipped shapefile for a large dataset, the GUI doesn't give you an option of where to save the dataset. So, if it is a long download like this dataset (Dataset | FWC GIS & Mapping Data Downloads), you have to wait until the large data download is complete before you can save it anywhere, and you might miss the prompt that shows up breifly at the bottom of the browser window.

Any suggestions would be helpful,

Chris Boland

Re: AGO Open Data download issues

DanielFenton1 — Wed, 17 Dec 2014 00:13:35 GMT

Hi Chris, let me answer these issues one at a time.

1. I've investigated and it's unclear what's happening here. I'm going to forward this along to our team for analysis.

2. This download is failing due to a known issue with ArcGIS Server. We have a workaround in place in Open Data, it should be deployed early next week.

3. I believe this is a browser specific setting. However, I have an idea for how we could give you the option to right click and save as, when you are retreiving large downloads. I'll talk to the team about getting that implemented.

I'll keep you updated here with regards to the 2 bugs and the enhancement. Let me know if you have any other questions.

Daniel

Re: AGO Open Data download issues

ChristopherBoland — Wed, 17 Dec 2014 14:25:36 GMT

Thanks Daniel.

I think on #3, you're right. It is probably browser specific. I am using Firefox 34.0.5. I do get the save as option with Internet Explorer, and I do not have Chrome installed on my machine. I also have not tried Safari.

Thanks again,

Chris

Re: AGO Open Data download issues

ChristopherBoland — Fri, 19 Dec 2014 18:30:31 GMT

Daniel,

I spoke with Andrew Stauffer over the phone yesterday and I think on #3 it is because my OpenData site had not been upgraded to the Bootstrap version yet. I believe once I upgrade my site, this concern will be addressed with the new "My Activity" download menu.

Thanks again for looking into this,

Chris

Re: AGO Open Data download issues

DanielFenton1 — Mon, 22 Dec 2014 17:31:08 GMT

Hey Chris,

We have resolved issue number 1. You should be able to download that file without any future problems.

We are still working on issue 2. The extremely high vertex count per feature is presenting some challenges in converting the data into a shapefile or kml extract. If these features were simplified, the issue would not liekely arise. We want people to be able to download original source data so we're going to keep at it and I'll let you know when there are any updates.

Re: AGO Open Data download issues

MiguelParedes — Wed, 14 Jan 2015 12:00:41 GMT

Greetings Daniel,

I have been observing an interesting issue with an Open Data portal, which is none of the issue reported by Christopher, but that is pretty baffling.

On a particular Open Data portal, I have found a dataset of Planning Applications. When I check the Table tab, it reports a total of 33,096 features. However, when I download the data, either as CSV or as a shapefile, the downloaded datasets have double that number of features, or 66192. Closer examination of the downloaded datasets shows that the features seem to have been duplicated.

Have you heard and / or are you aware of this issue happening? If so, is there anyway to fix it?

Many thanks in advance.

Re: AGO Open Data download issues

ChristopherBoland — Wed, 14 Jan 2015 13:56:34 GMT

@Miguel Paredes,

This issue has been reported on another geonet discussion (Open data downloads - duplicate records ). No answer has been provided yet.

Chris

Re: AGO Open Data download issues

MiguelParedes — Wed, 14 Jan 2015 14:18:48 GMT

Greetings Chris,

Many thanks for the link. As a matter of fact, it was Simon's site and the Planning Applications data set I was referring to. Interesting to find out that there are others, well, at least one more individual having the same issue.

Would be great to have a response from Esri on the issue, and how to resolve it.

Thanks again,

Miguel

Re: AGO Open Data download issues

DanielFenton1 — Wed, 14 Jan 2015 15:08:41 GMT

Hi Miguel,

I've fixed the dataset in question. We are having trouble reproducing this issue in our testing environment, but we are working on a fix nonetheless.

Daniel

Re: AGO Open Data download issues

MiguelParedes — Wed, 14 Jan 2015 15:23:10 GMT

Greetings Daniel,

That is brilliant! Thank you very much for this.

Questions arise, is this caused by the size of the dataset in question? Or is it something else?

In any case, much appreciate the effort.

Miguel

Re: AGO Open Data download issues

DanielFenton1 — Wed, 14 Jan 2015 16:10:28 GMT

Miguel, our best guess is that simultaneous requests for datasets that are not cached causes this.

Re: AGO Open Data download issues

MalcolmJ — Wed, 04 Mar 2015 19:47:10 GMT

Daniel Fenton

I am having the same problem as in issue 1 in the original post.

When trying to download a large dataset (25,000 polygon features) the progress will stay at 0% for a few minutes, then say "An error occurred while processing this dataset". I've been testing our site over the last week or so and find this occurs on this particular dataset most times I try download it. On other datasets is happens at various tiems but not so frequently.

This is the dataset here Dataset | Marlborough District Council Open Data

The data comes from this 10.2.2 service Layer: Property (ID: 36)

Re: AGO Open Data download issues

DanielFenton1 — Thu, 05 Mar 2015 19:30:39 GMT

Hi Malcolm,

I cleared the cache and the download seems to be working well. As of last night, you have a new button in the admin data report for your site which will allow you to clear the cache on your own. Clearing the cache is kind of the unplug it and plug it back in approach troubleshooting downloads. Works most of the time!

Re: AGO Open Data download issues

MalcolmJ — Thu, 05 Mar 2015 20:42:07 GMT

Thanks Daniel. That dataset is now downloading consistently without issues. The clear cache button is a great feature! Nice to have a way to fix this issue ourselves if it pops up.

Thanks heaps

Re: AGO Open Data download issues

DanielFenton1 — Fri, 06 Mar 2015 16:05:32 GMT

Glad you like it! It's feedbacks from folks like yourself on this forum that led to us implementing it. So please keep up the communication.

Daniel

Re: AGO Open Data download issues

NatalieHarris — Fri, 02 Feb 2018 17:18:01 GMT

I am also having this issue. We are uploading a layer of parcels to open data with 163,462 rows (polygons). We do this monthly. This month for some reason, when we download the parcels, the number of polygons is correct, but there are holes in the data where some polygons are no longer present. The data must be getting lost in the download of the zip file. What is going on? The data is located ArcGIS Hub . The January dataset is also looking like it's affected now. Please let me know what the issue is. Thank you.

Re: AGO Open Data download issues

DanielFenton1 — Fri, 02 Feb 2018 18:47:01 GMT

Hi Natalie,

I recommend opening up an issue with support so they can precisely scope the issue and further escalate to developers if necessary.

Daniel Fenton

ArcGIS Hub Team