Open data downloads - duplicate records

6035
17
01-09-2015 02:07 AM
SimonLutman
New Contributor II

I have a dataset with 30,910 records displayed in my ArcGIS Open Data portal. The Table tab indicates that it's 'Showing 30910 rows' but when I download the dataset, as either CSV or Shapefile, there are 60,000 records, mostly duplicate. I've updated the Index on the Data Report admin site and used different browsers, but still get the same result. Any suggestions?

See the offending planning applications dataset in the ArcGIS Open Data portal

Regards

Simon

Tags (1)
0 Kudos
17 Replies
CameronMcCormick
Occasional Contributor

Having the same issue with our data.  It also looks like it's happening with several of our data layers.  I've tried all the same things Simon, and had exactly the same result.

I've also tried it with an "old" style site and the "new" bootstrap style and it doesn't seem to make a difference, so we can probably mark that off the list of offenders.

Here is our example: Dataset | City of Chesapeake, VA Open Data Site

Cameron

0 Kudos
DanielFenton1
Occasional Contributor III

Cameron I just downloaded the dataset in question and did not see any duplication. It may have been a temporary issue. Have you seen this with any other datasets?

0 Kudos
CameronMcCormick
Occasional Contributor

Yeah, I just tried our parcel data again this morning, and it's still duplicating. Dataset | City of Chesapeake, VA Open Data Site , so is the 2035 Land Use Classes from my first reply.

I've cleared my IE browser cache and used Firefox with no cache and get exactly the same results.

It's downloading the parcels .zip (41.6MB) extremely fast.  Could it be some temporary internet file?

I just tried a "new" dataset, one I haven't downloaded before and it seems like it's 100% correct. Dataset | City of Chesapeake, VA Open Data Site


Cameron

0 Kudos
DanielFenton1
Occasional Contributor III

Cameron, I've resolved the issues with those two datasets. Let me know if they crop back up.

0 Kudos
CameronMcCormick
Occasional Contributor

Thanks Daniel!

Is there anything on our end we can do to prevent this from happening in the future?

0 Kudos
DanielFenton1
Occasional Contributor III

Unfortunately not.

0 Kudos
DanielFenton1
Occasional Contributor III

Hi Simon, I have fixed the dataset you were having issues with.


We have had trouble duplicating this issue in our test environment so solving it outright has been tricky. We're still working on it though. Please let me know if you run into this issue again with this or any of your other datasets.

SimonLutman
New Contributor II

Thanks for fixing this Daniel and thanks for your feedback Cameron McCormick. All seems well again

0 Kudos
CortDaniel1
New Contributor III

Daniel,  We are having the same issue with our Open Data website.  I assume the fix isn't global since I just downloaded the data set with the duplicates (Dataset | Pierce County WA Open GeoSpatial Data Portal (beta).‌ The Open Data site displays the correct number of records 51,668, but the downloaded shapefile has 84336.  We are hosting the data on Open Data, there is not a direct link to our database. 

Cort

0 Kudos