File Geodatabase downloading with .gdb extension

[CortDaniel1]

File geodatabases are downloading from our ArcGIS Hub website with .gdb file extension instead of the .zip extension.  The downloaded files are zip files.  Replacing the .gdb extension with .zip allows the FGDB to be extracted.   Is anyone else experiencing this problem?  Daniel Fenton‌ can you look into this?

Our website is https://gisdata-piercecowa.opendata.arcgis.com/.  

Thanks!

[CortDaniel1]

Rich,

We still see data discrepancies.  Have the code changes been pushed to the entire system? 

The zip code FGDB contained old data this morning when it was downloaded around 7:15 am.  We ran an update and at 7:40 am there is still no change.  

Other data was updated this morning.  It was checked around 7:00 am.

Address Points Updated at 4:41 am (point data)
URL: https://gisdata-piercecowa.opendata.arcgis.com/datasets/address-points
In house count: 387654
387695 shp old count
387695 fgdb
387695 csv
387,654 Website new count

Roads Updated at 6:02 am (line data)
URL: https://gisdata-piercecowa.opendata.arcgis.com/datasets/roads
In house count: 50992
50992 shp
50991 fgdb
50992 fgdb 2nd time downloaded
50992 csv
50,992 Website

Permits Updated at 6:36am (point data)
URL: https://gisdata-piercecowa.opendata.arcgis.com/datasets/permits-pierce-county
In house count: 635639
635238 shp
635238 fgdb
635573 csv
635,572 website

[RichGwozdz]

Hi Cort -

Thanks for reaching out again.  At least part of what your are experiencing may come from the delay in generating a fresh FGDB.  In an effort to deliver downloads quickly, the Opendata website follows the following flow when a file is requested for download:

1. Check to see if the file is already available for download

2. If file is available, check to see if the file is out-of-date relative to its hosted feature service; if it is outdate, begin generating a new file with the latest data. Note that this can take several minutes to hours depending on the size of the dataset.

3. Deliver the existing file, even if it is out of date, so that the user avoids long wait times for large downloads.  The fresh FGDB overwrites the out of date file as soon as it is available.  All downloads from that point on will use the fresh FGDB.

So in the case of larger downloads, the updated dataset may not be able to be downloaded for 10's of minutes to hours.

As of this writing, I find that downloaded FGDBs for your "Address Points", "Roads", and "Permits" have record counts that match what you noted for the website count (and the hosted service API).  So I think the discrepancy you reported above for these datasets was due to the new FGDB files not yet being completely generated; as such you were receiving the file on hand, which was out of date.

Regarding permits, I notice in your message that your in-house count doesn't match the number on the website.  It looks as though your in house data for Permits was not published properly.  The hosted service has a last edit date of Monday, April 15, 2019 6:32:33.723 PST.  That would explain the difference between your in-house count and website count.

Regarding Zipcodes, can you describe the changes you were expecting to see in the FGDB download?

[CortDaniel1]

Hi Rich,

Thanks for explaining the update process on your side. We can write a process to trigger the data download to general fresh FGDB, shapefiles, and csv files. The problem for our users is that not all data and references to the data are updated at the same time.

The Address points overview tab shows 387,654 records and the date is 4/16/2019. This is the same date the data update was run. The number of records matches the new data. When a user downloads the data as a shapefile, csv, and fgdb the records counts are all 387,695. This the old record count. The end user is left guessing if they have the most current data. Since the refresh time is not consistent, we cannot tell them exactly when the new data will be available for download.

The Roads data was updated 1 1/2 hours after the address data. After the FGDB was downloaded once, all references to the data record count were consistent. Are there better times to update data?

The permit data is updated an hour before it is updated in Open Data. The mxd has a field turned off that is not useful to anyone outside our agency. It is possible one of these things caused the data not to publish properly. We will look into changing process times and removing the field in another way. Is there a way to script a check that it did not publish properly?

The zip code change we expected to see in the FGDB are in the above screen shots. I downloaded the FGDB recently, and it does not show the boundary change. The shapefile does show the change.

[RichGwozdz]

Cort -

Let's take these one at a time and see if we can figure out what's happening.  When I visit the Address Points page (Address Points) right now and download the fgdb (https://opendata.arcgis.com/datasets/1501552c77d5462ea141c9bf68584f16_0.gdb?outSR=%7B%22latestWkid%2...), I receive a FGDB, which when opened displays 387,654 records.  That seems to be what you're expecting.  Can you confirm you get a different record count with a freshly triggered download?

[CortDaniel1]

Rich, 

I just downloaded the Address points data (fgdb, shapefile, and csv). 

The counts were:

FDGB: 387695

Shp: 387695

csv: 387695

Then I downloaded the same files using Chrome's Incognito window.  All three have the current data, 387,654.  

This may be a browser issue.  The problem is that if I am having this problem, then others will have it too.   We had a similar issue with websites we wrote years ago.  The solution was to append a cache buster GUID to the URL string.  The browser would then see the page as new and refresh it.  

[RichGwozdz]

Can you confirm this solves your issue with ZipCodes?  I'm not familiar enough with the zipcodes of Pierce County for your screen capture to aid me examining the downloaded data.

Regarding publishing problems with your MXD, I would post a new GeoNet issue specifically for that so that you can get the proper expertise - it's not specific to ArcGIS Hub.

[CortDaniel1]

Either going incognito solved the zip code problem or it finally update in Open Data.