Open data downloads - duplicate records

[SimonLutman]

I have a dataset with 30,910 records displayed in my ArcGIS Open Data portal. The Table tab indicates that it's 'Showing 30910 rows' but when I download the dataset, as either CSV or Shapefile, there are 60,000 records, mostly duplicate. I've updated the Index on the Data Report admin site and used different browsers, but still get the same result. Any suggestions?

See the offending planning applications dataset in the ArcGIS Open Data portal

Regards

Simon

[DanielFenton1]

Hi Cort, the fix is not global but I have resolved that individual dataset. It's an issue with our infrastructure, it does not matter where your data is hosted. We are testing some potential solutions now, but this is a slippery one for us.

[CortDaniel1]

Thanks Daniel!  If this happens again, is there a way I can fix this myself?   If not, who should I contact to have the issue fixed? I'd prefer not to have to post here to get the problem fixed.

Thanks!\

Cort

[DanielFenton1]

Cort, you can email me at dfenton@esri.com. I hope that we'll have this solved soon, but unfortunately at this time we don't have a way for you to fix this yourself.

[DanielFenton1]

All,

We put a fix in to our infrastructure that should resolve this issue entirely moving forward. There may still be some cached downloads that contain duplicate features. We will be running some tests to look for and destroy these caches, but if you have any issues in the meantime, please let me know and I will manually clear your site.

Daniel

Product Engineer | Open Data

[DanielFenton1]

Actually, it now looks like our caches are not expiring properly so downloads may be getting doubled overnight. Stay tuned, we are looking into this now...

[SunnyWisher]

So was this issue ever resolved or any fix determined?  I see that it was fixable but it doesn't appear to explain the solution.  For a newly loaded table I don't have this issue.  But when updating the records in the table (using either truncate and append OR Delete Rows and append) it duplicates the records.  When initially adding a new table that is not a problem.  any help is appreciated. 

[DanielFenton1]

Hi Sunny, the original issue was resolved 3 years ago. If you are seeing duplication it is certainly caused by something else. Have you opened a support ticket yet?

Daniel Fenton

Software Engineer | ArcGIS Hub

[SunnyWisher]

Well, thank you for your response. although it was about 4 months too late, I didn't notice it until 3 months later either.  I suspect the issue was related to publishing a spatial view within a map service, to get around that issue I simply increased the data creation and workflow to export the view to a feature class nightly. 

I did not submit a support request because that usually doesn't result in much but wasted time from my experience. Especially with the time it takes to get to someone who can actually help.  Plus not everyone has unlimited support options.

Thank you for your reply though, all the best.