Improvements people would like for Open Data

[MalcolmJ]

I thought I'd start a discussion about the improvements people would like for Open Data.

Two of my bigger issues were ticked off recently with the introduction of domain support and local projection for shapefile downloads, so thanks for that!

Here are two other things I'd quite like to see happen

1. Instead of filtering by the map view, I think it would be easier for users to draw an extent on the map of an area they are interested in. Could be a simple as a rectangle extent. For a good example of this, see here

2. I would like the "Updated" date to change when the underlying data is changed, not when the ArcGIS Online settings for that layer are updated. For example, the data in this layer is updated weekly, yet the "Updated" date says 3 months ago. This can be very misleading for users.

If anyone else has ideas for improvement, feel free to add them.

[DanielFenton1]

Hi Malcolm,

Thanks for posting your ideas!

As far as drawing areas, we're cooking up better support for image services so you'll see a lot easier ability to download areas your interested. For vector data, we haven't really thought about allowing the users to filter by a drawn rectangle. We'll ping this off a few other folks and see if there's broader support for that functionality.

We'd also love to have the updated date show the actual date of the data. It turns out to be a pretty tough problem, unfortunately. However, we've got some improvements coming down the line that will leverage editor tracking. The best thing you can do to help us with that is turn that feature on for all the layers you are serving in Open Data. Here's some help on that topic: ArcGIS Help 10.1

[MalcolmJ]

With drawing areas, I've used both methods through different GIS download websites. Filtering by map view in Open Data and by drawing a rectangle extent in https://data.linz.govt.nz and personally, I find the rectangle extent a lot easier to use. Others may disagree...

With updated date, the data we provide through Open Data is a copy of the main production data. Its mostly updated once a week using Delete and Append. So in our case, our updated data gets entirely replaced, so would get new OBJECTID's. Would a possibility to detect data updates be to use a combination of editor tracking and/or looking for new OBJECTID's?

[MalcolmJ]

I thought of two more improvements as well.

- When using aliases on a field in a dataset being served through Open Data, in the Open Data site the fields will display using the alias field name, but when you download, it will revert to the actual field name and ignore the alias. I think the downloaded data should match what's displayed in Open Data.

E.g. this layer uses an alias on the "Decision Date" field. The actual field name is AppIssDt, which is what is shown when the data is downloaded.

- It would be useful to be able to download as file gdb. In particular do get long field names without truncating them, as a shapefile does.

[DanPatterson_Retired]

geodatabase as an option yes...as the only source ... no, since open source data should be useable in OS software  and shapefiles is one of the major types

[JoshuaBixby]

Regarding file formats, I find it a bit humorous that many ckan-based sites like the Minnesota Geospatial Commons already offer downloads as file geodatabases and geopackages when Esr's Open Data site is stuck with KML and shape files.

[DanPatterson_Retired]

My point...offer them in a variety of formats, or one format that all software can read...

Minnesota's DNRGPS didn't read .gdb...

​EDIT  my bad...added since the last version

Upload/Download Waypoints,Tracks,Routes	Save as .gdb, .shp, .txt, etc

[AndrewTurner]

Thanks for the discussion on file formats

CKAN links to hand-generated static files while ArcGIS Open Data provides dynamic data conversion into common and easy to use formats by the broader data user community.

Our reasoning is that GIS Users can already export as File Geodatabase from online services, or data providers can include links to static FDGB files in the description where that is desired. This is essentially the same as required for the mentioned CKAN links.

We are considering ways to extend the download options to other formats while not overwhelming the end-user.

[JoshuaBixby]

Andrew Turner​, thanks for the information and insight.  I understand what you are saying from a technical perspective, but what makes sense from a technical perspective doesn't always make sense from a user perspective.  Data providers could provide a link to static FGDB in the Description, but why have one format accessible under the Details tab while the rest are under "Download Dataset"?  If there is a big blue button labeled "Download Dataset," I think most users will look there first, and possibly only there, for FGDB data.

From a Desktop GIS perspective, I would guess 50% or more of ArcGIS Open Data users are also ArcGIS Desktop users.  Why not support a format that works best for 50% or more of Desktop GIS users of ArcGIS Open Data?

From an open data perspective, GeoPackage is more open than shapefile.  I understand shapefiles are, or historically have been, ubiquitous, but making a standard publically available isn't the same as making an open standard.  Shapefiles might still be needed for legacy purposes, but let's embrace open formats and look more toward the future.

For a typical user, I don't think adding a couple additional download formats will be nearly as confusing as what those API links are or are not for. : )

[Lynchburg_GISDivision]

We'd also like to see support for FGDB downloads.