Updating Hosted Layer from Enterprise GDB

1236
6
05-18-2021 09:05 AM
Labels (1)
James_Armstrong
Occasional Contributor

Good Day All,

I hope I am posting this in the correct section.  I have an enterperise database (SQL) and ArcGIS Server (Portal) 10.8.  The organization would like to share out data and we have set up a open data site.  Using this open data site we have publishe out several datasets as hosted layers (using ArcGIS desktop).  Seems to work fine as the data is available to download in several formats. Data download options do not seem to work for Feature Layers that is published as referenced data. Of course for us, this would be ideal.

The issue arises when the question of data update is asked.  We were hoping that the way it was published that the data would be automatically updated, but as we find out this is not the case.  So it seems that updating the hosted Feature Layer is done manually. (assuming that the service definition is still present).

So Questions:

  • Are we missing or is there a method or process that the Hosted Feature Layer can be updated automatically from the Enterprise GDB?
  • Can non-hosted feature layer be set up to allow download options?
  • IS there any advantage in using ArcGIS Pro for this process?
  • Could this process (updating Hosted Feature Layers) be accomplished though model builder or python script?

Any other suggestions or ideas on this open data site issue would be appreciated.

James A.

0 Kudos
6 Replies
jcarlson
MVP Esteemed Contributor

Are you using the Enterpise version of Sites? Or are you using Hub?

In an Enterprise site, users can't download items unless they are able to own content items in your portal. Through Hub, they can download directly.

We do this by sharing our Open Data layers across to AGOL via a Collaboration that populates an Open Data group in AGOL and feeds into a Hub site there.

- Josh Carlson
Kendall County GIS
James_Armstrong
Occasional Contributor

Thanks for the insight.  I may be mistaken, but I believe that Enterprise sites allow for annonymous download now.  It was an issues, but a several weeks ago there was a fix....(cant recall but it may have been updating to 8.1.1.   The issue still remins that only as a hosted feature layer can the download be done via the enterprise sites.  Still you suggestion has some merit as it acutally may solve another somewhat realted issue we are having.   thanks again,  James

0 Kudos
jcarlson
MVP Esteemed Contributor

We just upgraded from 10.8.1, and it still didn't then. Haven't checked in 10.9 yet.

If you go the SDE → Hosted layer method, I do have a python script I could share that handles that very procedure. The trick there is catching deleted features.

- Josh Carlson
Kendall County GIS
0 Kudos
James_Armstrong
Occasional Contributor

Josh,  Thanks, I would appreciate the script if you don't mind sharing it.    I did go back in my notes and it seems that after publishing a Hosted Feature layer, the following steps can be taken to create an anonymous download setup. 

1.  Share the feature layer with everyone. 

2.  Share the FL  with the content group that is set up as the data sharing Enterprise Site.

3. Then, go to the site/application as a signed in Creator and download the formats that you want to have shared.  This act of doing it as a Creator somehow opens the downloads up for anonymous download.  I noted that it creates files of each format in the portal content, usually under the folder where the feature layer is located.  It does work!.

Hope this helps and again, please send the python script you mentioned.  I would be interested in checking it out..

James Armstrong  

0 Kudos
jcarlson
MVP Esteemed Contributor

Here's the bare bones of the script that you can modify. The full thing is full of peculiarities specific to our data, but this is the essential stuff.

Also, note that the hosted counterpart layer has a GUID field added to it.

## Modules
from arcgis import GIS
from arcgis.features import FeatureLayerCollection
import pandas as pd
import numpy as np

## Setup
# Portal Connection
user = input('Username: ')
password = getpass('Password: ')

gis = GIS('your-portal-url', user, password)

# Layers
service = FeatureLayerCollection('service-url/FeatureServer', gis)
hosted = FeatureLayerCollection('hosted-url/FeatureServer', gis)

## Update Process
# Query services to dataframes
s_df = service.layers[0].query(as_df=True).rename(columns={'globalid':'hosted_guid'}).set_index('hosted_guid', drop=False) 

h_df = hosted.layers[0].query(as_df=True, return_geometry=False, out_fields=['objectid', 'hosted_guid']).set_index('hosted_guid', drop=False)

# Merge the two
df = s_df.merge(h_df.loc[:, 'objectid'], left_index=True, right_index=True, how='outer', indicator=True).replace({np.nan:None})

# Apply updates to existing features (those in both service and hosted)
updt_df = df.loc[df['_merge'] == 'both'].reset_index(drop=True).astype({'objectid':'int'})

hosted.layers[0].edit_features(updates=updt_df.spatial.to_featureset())

# Add new features (those only in the service)
adds_df = df.loc[df['_merge'] == 'left_only'].reset_index(drop=True)

hosted.layers[0].edit_features(adds=adds_df.spatial.to_featureset())

# Delete features (those gone from service but still present in hosted)
deletes = df.loc[df['_merge'] == 'right_only'].reset_index(drop=True)['objectid'].astype('int').to_list()

hosted.layers[0].edit_features(deletes=deletes)

 

 In short, the comparison between the two services as dataframes can be used to identify adds, updates, and deletes.

For updates, we have a secondary query that uses the last_edited_date field to limit the features returned, to ensure that only features edited since the previous run of the notebook are actually updated.

We also put most of what I posted into a series of custom functions so that the global scope isn't getting messy with lots of dataframe objects. Then it's just custom_function(service_layer, hosted_layer).

But again, a lot of those extra details are quite specific to our dataset. The code above should be enough to start the process and modify it as needed.

- Josh Carlson
Kendall County GIS
0 Kudos
Todd_Metzler
Occasional Contributor III

Hello,

You are in luck with one exception.  The exception is download as FGDB option.

workflow:

1.  Maintain your data in RDBMS (SQL server).

2.  Register a read only connection to your SQL server DB with ArcGIS Enterprise.  Suggest a low permissioned (connect, select...) DB authenticated account for this.

3.  Publish a service to your ArcGIS Enterprise as feature service type.  Feature service type data must be hosted in EGDB like SQL server or ArcGIS Data Store.

4.  Add content with the service URL in your ArcGIS On Line Organization.

5.  Share to a group that has open data (aka hub) enabled.  You'll then see the content in open data (aka hub) with the download options except FGDB.

6.  When the content is updated in your SQL DB, the hosted feature layer will see those edits and your end users will too.

EXAMPLE: Hub Content sourced from ArcGIS Enterprise 

Todd

0 Kudos