Top 10 Questions and Answers from LTS - Getting Started with ArcGIS Open Data

[#1]

There was a Live Training Seminar on Getting Started with ArcGIS Open Data.  This was presented by Courtney Claessens and Daniel Fenton and was a great introduction to using and producing Open Data.  Below are some of the questions and answers from that Live Training Seminar (LTS).

1. Does using ArcGIS Open Data use the hosting organization’s ArcGIS Online Credits?

• No. Organizations are never charged any credits for set-up or usage of their sites. The only way credits may be consumed is if you opt to host your data is in ArcGIS Online. Credit usage is based on the size of data stored, not on usage or number of downloads.

2. Are hidden fields included when users download datasets?

• No. Fields that are hidden in ArcMap before a layer is published will neither show up in the Open Data application nor in any downloaded files.

Caution: Do not hide the ObjectID field as this will cause errors in the Open Data application.

3. What is the output projection and can it be changed?

• The output projection for the map within the Open Data application is Web Mercator (EPSG: 3857), the same projection used within ArcGIS Online maps. The output projection for file downloads and APIs (where applicable) is WGS 84 (EPSG: 4326). At this time output projections cannot be changed. If you would like to provide users downloads in additional formats, we recommend placing a link in the description text for the dataset.

4. Can one organization have multiple Open Data sites?

• Yes. An organization can have as many Open Data sites as they desire. For example a state-level organization could create Open Data sites for each county or a city could create an event-based (e.g. hackathon) site for a weekend and then take it down.

5. Can you upload and share data that has no geospatial component?

• Yes. Currently Open Data supports tables published in services that run local on ArcGIS for Server, or hosted in ArcGIS Online. You can also upload and share CSV files in ArcGIS Online that are up to 5mb and comma separated.

Advanced users can leverage Esri’s Open Source Project KOOP to access non-spatial data from other sources, and provide GeoServices REST API to the data. There are currently connectors for data stored in Github as well as in Socrata. [AS1] (see more information here).

6. If my organization is using a different coordinate system, do we need to re-project it to Web Mercator before using it in Open Data?

• No. ArcGIS Open Data handles data re-projection for you in the background.

7. Should I re-project my data to Web Mercator before publishing in Open Data?

• You do not have to, but it can help. Re-projecting your data to Web Mercator can improve performance when viewing the map in Open Data, since the data needs to be in Web Mercator to be display on the grey basemap. When your data is in a local projection, it is requested from the application in Web Mercator and the server may need additional processing time to re-project the data and provide a valid response.

Note: Open Data will soon support custom base maps in local projections. In that case, if the projection of the data matches the projection of your basemap there will be no performance loss from re-projection.

8. Is there an Open Data site gallery to let users know what agencies have public Open Data sites?

• Yes. There is an Open Data showcase located at http://dc.esri.com/showcase that shows handpicked Open Data sites that have been made public. Additionally, you can always visit http://opendata.arcgis.com/explore.json to download a file describing all Open Data sites that are currently public.

9. Do you integrate with other platforms like CKAN?

• Yes. All Open Data sites produce a catalog that CKAN can ‘harvest’. To access this catalog simply append /data.json to the end of the URL for your Open Data site. Then, you will need to configure your CKAN instance to read that file. See some information about that here (https://github.com/ckan/ckanext-dcat).

10. If I am hosting data on my own ArcGIS Server, what is the best way to serve them for use in Open Data.

• If you are setting up services with the exclusive purpose of serving them on an Open Data site we recommend you create a single map service then register each individual layer as an item in ArcGIS Online. With a single hosted map service, you can more easily control resource utilization on your server. With each layer registered as an individual item, you have control over metadata related to each layer in ArcGIS Online.

[Suzanne-Boden]

Great info. The seminar recording can be accessed at: http://training.esri.com/gateway/index.cfm?fa=catalog.webCourseDetail&courseID=2827.

[Lars_Kristian_EngelsbyHansen]

Great information.

According to number 9, CKAN can harvest from ArcGIS Open Data. But is there any opportunity for doing it the other way around?   Harvest from CKAN to ArcGIS Open Data? James Gough‌

[DominikSchlarmann]

Hey @LarsKEHinformi-dk-esridist

As far as I have understood you in the right way, there is also the possibility to harvest CKAN sites, wrap data into FeatureServices and make them accessible in ArcGIS Online...the last step has to be done manually, but I am sure that the system can be extended to do this automatically.

Please check koop for more information: Esri/koop · GitHub

I've tested the system in the past weeks and it works, apart from some limitations, without bigger problems.

There is also another system, which does a harvesting on CKAN-Sites. Please check the OpenDataBridge for more information: EsriDE/OpenDataBridge · GitHub

All the best,

Dominik

[Lars_Kristian_EngelsbyHansen]

Yeah, I have already used the Koop GeoJson to to Feature service, but I think its a too manual process. In one case I had more than 80 GeoJson's from this site: Velkommen - Open Data København it took a while to get them into this Open Data site: Home | KK data (download GeoJsons -> upload them to Github -> add them as items in ArcGIS Online)

The "harvest" from ArcGIS Open Data to CKAN seems a lot easier than the other way around but I will try the EsriDE/OpenDataBridge · GitHub , thanks for the heads-up..

[DominikSchlarmann]

Hey Lars Kristian Engelsby Hansen‌,

You're welcome

Just a few lines about the koop-stuff.

There is already an existing koop-ckan provider, which does a major part of the stuff, you are searching for. Once you have the ckan-datasets ID, which is the last part of the URL when you are reviewing a dataset on the portal (e.g. it is "Parkomater" for the link Parkomater - Open Data København), you can access this dataset via koop. The dataset will be transformed to a geojson file/stream and also to an ESRI FeatureService, which is the part you'll need.

With the FeatureService-Link you can add this item to your ArcGIS Online subscription, share it with your ArcGIS OpenData Group and you're done.

I'm quite sure that you can perform the manual part of registering the link as an item to your ArcGIS Online Account and share it with the your ArcGIS OpenData Group automatically using the ESRI REST API, which is able to do this stuff via REST requests....but this is not a part of koop.

Unfortunately the existing koop-demo has not registered the ckan-portal, which you would need, so I can not show you a working example with one data.kk.dk item using the "official demo server".

Alternatively I have set up your ckan-site on my demo server, which you can access here (please note that the server won't be up all the time): My Example Server using your ckan portal.

Following the link, you will notice that this koop instance has already processed a dataset and transformed it into a geojson. Unfortunately the .csv format (which is until now the one and only format, which can be handled by koop-ckan) has not the needed structure to built up a valid geojson. You might notice that the geometry-array is null, but the coordinates are stored in the properties. This is caused by the fact that koop searches for columns called longitude/latitude, long/lat or x/y.

The coordinates in the datakk datasets are all stored in one column called "wkb_geometry" including its geometry type....some adoptions would be needed to fix that issue.

Anyway...using the following URL you should be able to add the dataset to your ArcGIS Online account and then to your ArcGIS OpenData Portal: My Example-Server-FeatureServer.

The data won't be displayed on the map (because of the wrong schema), but I assume you will be able to list it and download it via the ArcGIS OpenData mechanisms.

All the best,

Dominik

[Lars_Kristian_EngelsbyHansen]

Wow - thats great - Thanks alot Dominik ! And thanks for your time in answaring my questions.

I will look into it as soon as posible (read TODAY!!) 🙂

[AdamJoseph1]

if you do not have an ArcGIS server, you can still serve out feature services from Cartoview, install cartoview upload your shapefiles and follow step 10