Feature service which has no datasets?

[EricHanson]

I have been having some issues with open data lately. 

I am using a map service from my own ArcGIS Server.  I added items for each layer of the map service as individual items in AGOL as an item from the web with the service url for each layer.  I then shared the items (which are public) with my open data group which is Public and designated as available for use in Open Data.

In my open data site (which is currently Private for testing), I then added this group in the Manage Data tab.

A couple of weeks ago, this worked fine and all the data was showing properly.  But now when I view the Data Report, it says “feature service which has no datasets” and in the Manage Sites page, it lists that the Open Data site has items but Datasets are n/a even though when I view the items in AGOL or query the layers in the map service, they work fine.

Has anyone else seen this issue lately or know what the problem/solution is?  Is this related to the recent update?  FYI – AGOL hosted services work just fine.

Thank you!

[ChristopherBoland]

Hi Daniel,

I checked this morning and it appears that all datasets are now working in open data.  It may be that the indexing is just taking about 24 hours or so from when the item was created.

Chris

[DanielFenton1]

Glad to hear it, Chris. We're still working through those issues with harvest volume. Once those are resolved things will return to normal speed.

[DanielFenton1]

Hey guys, we have dealt with the harvesting backlog. If unsharing and sharing datasets doesn't solve any lingering problems please let me know.

cc Christopher Boland‌

[TobiasFimpel1]

Chris, Daniel, Eric:

Many thanks for haring your information here. I really appreciate it. I am in the same situation and so far making the item private and the public again seems to work - do you have any info on whether this workaround always works, or whether it doesn't work for some cases? Have you found it reliable enough to make your sites public yet?

Also do you know of a place where I can read up on how the "data harvesting" works? What is ArcGIS Online doing under the hood?

Thanks again,

Tobias

[DanielFenton1]

Hi Tobias,

This reason this works is because unsharing the ArcGIS item will completely remove it from our database and reharvest the item completely. This is as opposed to when there is a change made to an ArcGIS item which will trigger an update only.  It should always work unless there is a problem with the item itself or some temporary issue with our harvesters.

Harvesting is kicked off when:

1. A new public item is shared in an open data group

2. A non-public item in an open data group is made public

3. A change is made to an item that is in an open data group and is public

4. A group that was not previously enabled for open data is enabled for open data

For map services, under the hood our harvesters:

1. fetch the arcgis online item

2. fetch the map service url

3. fetch all the child layers

4. conduct statistics on all the fields of each layer

5. add the datasets to the open data index

The time it takes to conduct step 4 is a function of [the number of layers, the size of each layer (i.e. feature count), the number of attributes of each later]. And that can cause delays in items showing up in the index.

We have literally hundreds of customers who have found harvesting stable enough to launch their sites. You can check out over 100 of them at dc.esri.com/showcase .

I hope you find this information useful. Please let me know if you have any other questions.

[TobiasFimpel1]

Extremely helpful information, Daniel. Thank you for taking the time. I have a follow up question (?how to move ti to a new thread?):

In the documentation that I have read so far I couldn't find any best practices for configuring ArcGIS Server map/feature services for optimum performance with ArcGIS Online Open Data. Do you have any recommendations, or does it really not matter as long as the querying capability is enabled? (Min/max instances, high/low isolation, max. # features returned, service timeout, etc....) 

Thanks again,

Tobias

[EricHanson]

Update:

I have been able to get other organizations map services to work with my open data site, but I still cannot get my map services to work.  These services are publicly accessible as they do work in public applications, but yet they do not seem to get recognized by open data.

Anyone have any suggestions?  Could this be a setting on the ArcGIS Server side when creating the map services or something to do with the ArcGIS Server infrastructure that would not allow the services to be recognized?

I had asked this questions to Esri Customer Service a few weeks ago, but there was limited response to that question (What are the reasons a map service would not be recognized by the open data site?).

Thanks!

[DanielFenton1]

Eric, would you mind sharing the ArcGIS online item as well as the URL to the map service? I'll take a look and diagnose where the problem lies.

[EricHanson]

Hi Daniel.

Here is an item I have tried and the corresponding URL.

Thanks for your help!

[DanielFenton1]

Eric, can you please make sure that item is public? I'm unable to view it.