topic Re: Open data downloads - duplicate records in ArcGIS Hub Questions

Open data downloads - duplicate records

SimonLutman — Fri, 09 Jan 2015 10:07:01 GMT

I have a dataset with 30,910 records displayed in my ArcGIS Open Data portal. The Table tab indicates that it's 'Showing 30910 rows' but when I download the dataset, as either CSV or Shapefile, there are 60,000 records, mostly duplicate. I've updated the Index on the Data Report admin site and used different browsers, but still get the same result. Any suggestions?

See the offending planning applications dataset in the ArcGIS Open Data portal

Regards

Simon

Re: Open data downloads - duplicate records

CameronMcCormick — Tue, 13 Jan 2015 12:37:47 GMT

Having the same issue with our data. It also looks like it's happening with several of our data layers. I've tried all the same things Simon, and had exactly the same result.

I've also tried it with an "old" style site and the "new" bootstrap style and it doesn't seem to make a difference, so we can probably mark that off the list of offenders.

Here is our example: Dataset | City of Chesapeake, VA Open Data Site

Cameron

Re: Open data downloads - duplicate records

DanielFenton1 — Wed, 14 Jan 2015 15:11:33 GMT

Hi Simon, I have fixed the dataset you were having issues with.

We have had trouble duplicating this issue in our test environment so solving it outright has been tricky. We're still working on it though. Please let me know if you run into this issue again with this or any of your other datasets.

Re: Open data downloads - duplicate records

DanielFenton1 — Wed, 14 Jan 2015 15:12:02 GMT

Cameron I just downloaded the dataset in question and did not see any duplication. It may have been a temporary issue. Have you seen this with any other datasets?

Re: Open data downloads - duplicate records

SimonLutman — Wed, 14 Jan 2015 15:55:04 GMT

Thanks for fixing this Daniel and thanks for your feedback Cameron McCormick. All seems well again

Re: Open data downloads - duplicate records

CameronMcCormick — Wed, 14 Jan 2015 16:14:32 GMT

Yeah, I just tried our parcel data again this morning, and it's still duplicating. Dataset | City of Chesapeake, VA Open Data Site , so is the 2035 Land Use Classes from my first reply.

I've cleared my IE browser cache and used Firefox with no cache and get exactly the same results.

It's downloading the parcels .zip (41.6MB) extremely fast. Could it be some temporary internet file?

I just tried a "new" dataset, one I haven't downloaded before and it seems like it's 100% correct. Dataset | City of Chesapeake, VA Open Data Site

Cameron

Re: Open data downloads - duplicate records

CortDaniel1 — Wed, 14 Jan 2015 18:27:50 GMT

Daniel, We are having the same issue with our Open Data website. I assume the fix isn't global since I just downloaded the data set with the duplicates (Dataset | Pierce County WA Open GeoSpatial Data Portal (beta).‌ The Open Data site displays the correct number of records 51,668, but the downloaded shapefile has 84336. We are hosting the data on Open Data, there is not a direct link to our database.

Cort

Re: Open data downloads - duplicate records

DanielFenton1 — Wed, 14 Jan 2015 20:04:56 GMT

Hi Cort, the fix is not global but I have resolved that individual dataset. It's an issue with our infrastructure, it does not matter where your data is hosted. We are testing some potential solutions now, but this is a slippery one for us.

Re: Open data downloads - duplicate records

DanielFenton1 — Wed, 14 Jan 2015 20:18:11 GMT

Cameron, I've resolved the issues with those two datasets. Let me know if they crop back up.

Re: Open data downloads - duplicate records

CameronMcCormick — Wed, 14 Jan 2015 20:26:48 GMT

Thanks Daniel!

Is there anything on our end we can do to prevent this from happening in the future?

Re: Open data downloads - duplicate records

DanielFenton1 — Wed, 14 Jan 2015 20:36:05 GMT

Unfortunately not.

Re: Open data downloads - duplicate records

CortDaniel1 — Wed, 14 Jan 2015 20:39:40 GMT

Thanks Daniel! If this happens again, is there a way I can fix this myself? If not, who should I contact to have the issue fixed? I'd prefer not to have to post here to get the problem fixed.

Thanks!\

Cort

Re: Open data downloads - duplicate records

DanielFenton1 — Wed, 14 Jan 2015 20:45:46 GMT

Cort, you can email me at dfenton@esri.com. I hope that we'll have this solved soon, but unfortunately at this time we don't have a way for you to fix this yourself.

Re: Open data downloads - duplicate records

DanielFenton1 — Tue, 27 Jan 2015 16:35:53 GMT

All,

We put a fix in to our infrastructure that should resolve this issue entirely moving forward. There may still be some cached downloads that contain duplicate features. We will be running some tests to look for and destroy these caches, but if you have any issues in the meantime, please let me know and I will manually clear your site.

Daniel

Product Engineer | Open Data

Re: Open data downloads - duplicate records

DanielFenton1 — Tue, 27 Jan 2015 18:55:59 GMT

Actually, it now looks like our caches are not expiring properly so downloads may be getting doubled overnight. Stay tuned, we are looking into this now...

Re: Open data downloads - duplicate records

SunnyWisher — Fri, 06 Apr 2018 13:29:53 GMT

So was this issue ever resolved or any fix determined? I see that it was fixable but it doesn't appear to explain the solution. For a newly loaded table I don't have this issue. But when updating the records in the table (using either truncate and append OR Delete Rows and append) it duplicates the records. When initially adding a new table that is not a problem. any help is appreciated.

Re: Open data downloads - duplicate records

DanielFenton1 — Tue, 24 Jul 2018 19:51:48 GMT

Hi Sunny, the original issue was resolved 3 years ago. If you are seeing duplication it is certainly caused by something else. Have you opened a support ticket yet?

Daniel Fenton

Software Engineer | ArcGIS Hub

Re: Open data downloads - duplicate records

SunnyWisher — Wed, 24 Oct 2018 15:10:35 GMT

Well, thank you for your response. although it was about 4 months too late, I didn't notice it until 3 months later either. I suspect the issue was related to publishing a spatial view within a map service, to get around that issue I simply increased the data creation and workflow to export the view to a feature class nightly.

I did not submit a support request because that usually doesn't result in much but wasted time from my experience. Especially with the time it takes to get to someone who can actually help. Plus not everyone has unlimited support options.

Thank you for your reply though, all the best.