topic Re: Improvements people would like for Open Data in ArcGIS Hub Questions

Improvements people would like for Open Data

MalcolmJ — Mon, 18 May 2015 04:11:44 GMT

I thought I'd start a discussion about the improvements people would like for Open Data.

Two of my bigger issues were ticked off recently with the introduction of domain support and local projection for shapefile downloads, so thanks for that!

Here are two other things I'd quite like to see happen

1. Instead of filtering by the map view, I think it would be easier for users to draw an extent on the map of an area they are interested in. Could be a simple as a rectangle extent. For a good example of this, see here

2. I would like the "Updated" date to change when the underlying data is changed, not when the ArcGIS Online settings for that layer are updated. For example, the data in this layer is updated weekly, yet the "Updated" date says 3 months ago. This can be very misleading for users.

If anyone else has ideas for improvement, feel free to add them.

Re: Improvements people would like for Open Data

DanielFenton1 — Fri, 22 May 2015 15:04:02 GMT

Hi Malcolm,

Thanks for posting your ideas!

As far as drawing areas, we're cooking up better support for image services so you'll see a lot easier ability to download areas your interested. For vector data, we haven't really thought about allowing the users to filter by a drawn rectangle. We'll ping this off a few other folks and see if there's broader support for that functionality.

We'd also love to have the updated date show the actual date of the data. It turns out to be a pretty tough problem, unfortunately. However, we've got some improvements coming down the line that will leverage editor tracking. The best thing you can do to help us with that is turn that feature on for all the layers you are serving in Open Data. Here's some help on that topic: ArcGIS Help 10.1

Re: Improvements people would like for Open Data

MalcolmJ — Sun, 24 May 2015 20:54:21 GMT

With drawing areas, I've used both methods through different GIS download websites. Filtering by map view in Open Data and by drawing a rectangle extent in https://data.linz.govt.nz and personally, I find the rectangle extent a lot easier to use. Others may disagree...

With updated date, the data we provide through Open Data is a copy of the main production data. Its mostly updated once a week using Delete and Append. So in our case, our updated data gets entirely replaced, so would get new OBJECTID's. Would a possibility to detect data updates be to use a combination of editor tracking and/or looking for new OBJECTID's?

Re: Improvements people would like for Open Data

MalcolmJ — Mon, 25 May 2015 20:27:53 GMT

I thought of two more improvements as well.

- When using aliases on a field in a dataset being served through Open Data, in the Open Data site the fields will display using the alias field name, but when you download, it will revert to the actual field name and ignore the alias. I think the downloaded data should match what's displayed in Open Data.

E.g. this layer uses an alias on the "Decision Date" field. The actual field name is AppIssDt, which is what is shown when the data is downloaded.

- It would be useful to be able to download as file gdb. In particular do get long field names without truncating them, as a shapefile does.

Re: Improvements people would like for Open Data

DanPatterson_Retired — Tue, 26 May 2015 00:10:36 GMT

geodatabase as an option yes...as the only source ... no, since open source data should be useable in OS software and shapefiles is one of the major types

Re: Improvements people would like for Open Data

JoshuaBixby — Wed, 27 May 2015 00:29:21 GMT

Regarding file formats, I find it a bit humorous that many ckan-based sites like the Minnesota Geospatial Commons already offer downloads as file geodatabases and geopackages when Esr's Open Data site is stuck with KML and shape files.

Re: Improvements people would like for Open Data

DanPatterson_Retired — Wed, 27 May 2015 00:37:57 GMT

My point...offer them in a variety of formats, or one format that all software can read...

Minnesota's DNRGPS didn't read .gdb...

EDIT my bad...added since the last version

Upload/Download Waypoints,Tracks,Routes

Save as .gdb, .shp, .txt, etc

Re: Improvements people would like for Open Data

AndrewTurner — Wed, 27 May 2015 01:59:16 GMT

Thanks for the discussion on file formats

CKAN links to hand-generated static files while ArcGIS Open Data provides dynamic data conversion into common and easy to use formats by the broader data user community.

Our reasoning is that GIS Users can already export as File Geodatabase from online services, or data providers can include links to static FDGB files in the description where that is desired. This is essentially the same as required for the mentioned CKAN links.

We are considering ways to extend the download options to other formats while not overwhelming the end-user.

Re: Improvements people would like for Open Data

JoshuaBixby — Wed, 27 May 2015 04:26:43 GMT

Andrew Turner, thanks for the information and insight. I understand what you are saying from a technical perspective, but what makes sense from a technical perspective doesn't always make sense from a user perspective. Data providers could provide a link to static FGDB in the Description, but why have one format accessible under the Details tab while the rest are under "Download Dataset"? If there is a big blue button labeled "Download Dataset," I think most users will look there first, and possibly only there, for FGDB data.

From a Desktop GIS perspective, I would guess 50% or more of ArcGIS Open Data users are also ArcGIS Desktop users. Why not support a format that works best for 50% or more of Desktop GIS users of ArcGIS Open Data?

From an open data perspective, GeoPackage is more open than shapefile. I understand shapefiles are, or historically have been, ubiquitous, but making a standard publically available isn't the same as making an open standard. Shapefiles might still be needed for legacy purposes, but let's embrace open formats and look more toward the future.

For a typical user, I don't think adding a couple additional download formats will be nearly as confusing as what those API links are or are not for. : )

Re: Improvements people would like for Open Data

ChristopherBoland — Wed, 27 May 2015 13:00:33 GMT

I presented my agency's Open Data site (geodata.myfwc.com) to all of the State of Florida Agency GIS Heads and one of their main requests was the select vector data by user drawn polygon. In particular, many of Florida's agency's have to query permit locations, water sampling wells, emergency facilities, etc. within a certain proximity of an event.

Re: Improvements people would like for Open Data

CourtneyClaessens — Tue, 11 Aug 2015 19:42:41 GMT

Thanks for the feedback, Christopher. We have this task on our backlog - good to know people are actively asking for it!

Re: Improvements people would like for Open Data

TimMinter — Fri, 14 Aug 2015 16:38:28 GMT

I'd like to see improvement in ArcGIS Online and Open Data providing full support for and integration with the geodatabase structure. That would be sweet .

There are a lot of variations at present on which geodatabase data types can be ingested into ArcGIS Online, then fully used by ArcGIS Online apps, app templates, app builders, app studios, and Open Data capabilities.

Deficiencies I have experienced and investigated recently are:

Subtypes and domains
Many-to-many relationship classes
Metadata

My story:

As a data manager for an organization with open data sharing obligations, I need to upload our geodatabase structure to ArcGIS Online and share it via Open Data in a way that respects and fully utilizes the on-premise source geodatabase structure so that I can meet our obligation to share our data with a minimum of effort, thus allowing me to focus more time and effort on core business activities.

Two recent experiences, not going into detail or completeness here:

upload fGDB to AGO, specify to publish a feature service. The feature service is published, but an awful lot of fGDB parts are stripped out. Download the the fGDB and discover that the parts really are gone. It's not just that the service ignored the parts. It appears that the parts aren't there.
publish geodatabase-sourced layer from ArcMap or ArcGIS Pro 1.1 to AGO, specify feature service. ArcMap 10.3.1 is the only version that gets subtypes and domains right in the resulting service. cool. However, now the default map viewer table tool and Open Data don't understand the subtypes and domains. Interestingly, the default map viewer popup gets it and presents results properly.
I've noticed some mis-representation around values stored in date fields when using hosted feature services as a source. I have not investigated, but my immediate suspicions point toward the possibility that some SQL Server queries might be experiencing the time data rounding issue.

Re: Improvements people would like for Open Data

Lynchburg_GISDivision — Tue, 18 Aug 2015 15:48:13 GMT

We'd also like to see support for FGDB downloads.

Re: Improvements people would like for Open Data

MarcWeinshenker1 — Tue, 25 Aug 2015 16:44:37 GMT

I like the ability in the Site Manager to create many sites. It's useful for working out different configuration options. But, when I get a baseline configuration I want to expand on, there seems to be no way to make copy of it or import it when creating a new site, so I am recommending the ability to copy and/or import sites.

Re: Improvements people would like for Open Data

RamiAlouta2 — Tue, 22 Sep 2015 08:54:25 GMT

Hi this is Rami from WHO,

I also have some points to add concerning theOpen Data Site, and they are as the follow:

1 Requesting
ability be considered to allow the scroll button on the mouse to additionally
control zooming.

2 Requesting
ability be considered to allow for the definition of custom zoom levels.

3 Requesting
ability be considered to allow for transparency to be set when multiple datasets
are available in one vicinity to make the outlines easier to view.

Re: Improvements people would like for Open Data

RebeccaStrauch__GISP — Thu, 22 Oct 2015 02:52:00 GMT

We are planning on using our local ArcGIS Server services on the OpenData site. I have a few test services working. However, when we do maintenance or shutdown a service, the Tables and Chart tabs show an error.

Not very informative to a user. It would be nice is OpenData could recognize when a service/URL is not available, and place text "currently not available" or some kind of symbol like a

symbol over the item on the gallery page....or at least have the error message say "Service is currently offline".

Is this something that can be considered in the future? Is there anyway to at leat customize the error message for items that are from services?

Re: Improvements people would like for Open Data

MalcolmJ — Sun, 08 Nov 2015 19:20:53 GMT

Something I have been thinking about lately that would be handy would be the ability to set custom URLs for datasets.

So instead of a link like this

http://data.marlborough.opendata.arcgis.com/datasets/05f33314475b495fa2eba9555baf317b_4

It would be cool if I could manually set it to something like this in the admin.

http://data.marlborough.opendata.arcgis.com/datasets/busroutes

This would make URLs more meaningful if sent as a link and embedded links from other webpages easier to maintain.

Re: Improvements people would like for Open Data

JamiDennis — Fri, 13 Nov 2015 23:12:52 GMT

Can we talk about the charting features? They are really elementary. I'd like to be able to at least chart multiple variables from the attribute table - and be able to say which goes on which access. I admit I'm still learning my way around the features, so someone tell me if this option exists. But really all I see is you select only one attribute and the charting tool will create its own axis and chart just that one feature.

more control over what to chart would be huge for my users.

Re: Improvements people would like for Open Data

ANRGIS — Wed, 18 Nov 2015 14:12:47 GMT

I'd like to echo a couple of the requests made initially:

1. The ability to export via a user defined polygon

2. Ability to download a FGDB

Functionality that I don't believe exists yet, but would be very helpful would be the ability to send URL parameters to our open data site to specify certain things like:

1. The layer by name

2. The extent to filter from

3. Queries for filtering

4. File format to download

5. etc.

This would be helpful for some organizations that use ArcGIS Online or any other web platform for mapping, the ability to provide a link to the end user to download a particular layer they may be interacting with in another application. For instance, they may be mapping parcels in a particular town. If a user identifies, or queries this in another application, it would be nice to provide the user with a link to our open data site that would select parcels, zoom to the same extent, and start downloading the data in a new window.

This could also simply function as a service to extract this information from an endpoint the Open Data publishes when your site is created/managed.

Re: Improvements people would like for Open Data

NinaRihn — Wed, 10 May 2017 19:22:24 GMT

Can we revive this discussion? there are still some things in here that would greatly improve Open Data that haven't been done yet. My top few:

1. ability to filter a dataset using a polygon rather than that wide basemap

2. ability to copy an existing Open data site (I would like to be able to do my edits in the non-production site)

3. ability to enable File geodb download without hosting the feature service on AGOL

4. ability to zoom in on the basemap using the mouse scroll button (not just +- buttons)

5 ability to have the exports use the field alias rather that the field name (to help mitigate the shapefile field truncation issue)