Updating Hosted Layer from Enterprise GDB

[James_Armstrong]

Good Day All,

I hope I am posting this in the correct section.  I have an enterperise database (SQL) and ArcGIS Server (Portal) 10.8.  The organization would like to share out data and we have set up a open data site.  Using this open data site we have publishe out several datasets as hosted layers (using ArcGIS desktop).  Seems to work fine as the data is available to download in several formats. Data download options do not seem to work for Feature Layers that is published as referenced data. Of course for us, this would be ideal.

The issue arises when the question of data update is asked.  We were hoping that the way it was published that the data would be automatically updated, but as we find out this is not the case.  So it seems that updating the hosted Feature Layer is done manually. (assuming that the service definition is still present).

So Questions:

• Are we missing or is there a method or process that the Hosted Feature Layer can be updated automatically from the Enterprise GDB?
• Can non-hosted feature layer be set up to allow download options?
• IS there any advantage in using ArcGIS Pro for this process?
• Could this process (updating Hosted Feature Layers) be accomplished though model builder or python script?

Any other suggestions or ideas on this open data site issue would be appreciated.

James A.

[jcarlson]

Are you using the Enterpise version of Sites? Or are you using Hub?

In an Enterprise site, users can't download items unless they are able to own content items in your portal. Through Hub, they can download directly.

We do this by sharing our Open Data layers across to AGOL via a Collaboration that populates an Open Data group in AGOL and feeds into a Hub site there.

[James_Armstrong]

Thanks for the insight.  I may be mistaken, but I believe that Enterprise sites allow for annonymous download now.  It was an issues, but a several weeks ago there was a fix....(cant recall but it may have been updating to 8.1.1.   The issue still remins that only as a hosted feature layer can the download be done via the enterprise sites.  Still you suggestion has some merit as it acutally may solve another somewhat realted issue we are having.   thanks again,  James

[jcarlson]

We just upgraded from 10.8.1, and it still didn't then. Haven't checked in 10.9 yet.

If you go the SDE → Hosted layer method, I do have a python script I could share that handles that very procedure. The trick there is catching deleted features.

[James_Armstrong]

Josh,  Thanks, I would appreciate the script if you don't mind sharing it.    I did go back in my notes and it seems that after publishing a Hosted Feature layer, the following steps can be taken to create an anonymous download setup. 

1.  Share the feature layer with everyone. 

2.  Share the FL  with the content group that is set up as the data sharing Enterprise Site.

3. Then, go to the site/application as a signed in Creator and download the formats that you want to have shared.  This act of doing it as a Creator somehow opens the downloads up for anonymous download.  I noted that it creates files of each format in the portal content, usually under the folder where the feature layer is located.  It does work!.

Hope this helps and again, please send the python script you mentioned.  I would be interested in checking it out..

James Armstrong  

[jcarlson]

Here's the bare bones of the script that you can modify. The full thing is full of peculiarities specific to our data, but this is the essential stuff.

Also, note that the hosted counterpart layer has a GUID field added to it.

## Modules
from arcgis import GIS
from arcgis.features import FeatureLayerCollection
import pandas as pd
import numpy as np

## Setup
# Portal Connection
user = input('Username: ')
password = getpass('Password: ')

gis = GIS('your-portal-url', user, password)

# Layers
service = FeatureLayerCollection('service-url/FeatureServer', gis)
hosted = FeatureLayerCollection('hosted-url/FeatureServer', gis)

## Update Process
# Query services to dataframes
s_df = service.layers[0].query(as_df=True).rename(columns={'globalid':'hosted_guid'}).set_index('hosted_guid', drop=False) 

h_df = hosted.layers[0].query(as_df=True, return_geometry=False, out_fields=['objectid', 'hosted_guid']).set_index('hosted_guid', drop=False)

# Merge the two
df = s_df.merge(h_df.loc[:, 'objectid'], left_index=True, right_index=True, how='outer', indicator=True).replace({np.nan:None})

# Apply updates to existing features (those in both service and hosted)
updt_df = df.loc[df['_merge'] == 'both'].reset_index(drop=True).astype({'objectid':'int'})

hosted.layers[0].edit_features(updates=updt_df.spatial.to_featureset())

# Add new features (those only in the service)
adds_df = df.loc[df['_merge'] == 'left_only'].reset_index(drop=True)

hosted.layers[0].edit_features(adds=adds_df.spatial.to_featureset())

# Delete features (those gone from service but still present in hosted)
deletes = df.loc[df['_merge'] == 'right_only'].reset_index(drop=True)['objectid'].astype('int').to_list()

hosted.layers[0].edit_features(deletes=deletes)

 In short, the comparison between the two services as dataframes can be used to identify adds, updates, and deletes.

For updates, we have a secondary query that uses the last_edited_date field to limit the features returned, to ensure that only features edited since the previous run of the notebook are actually updated.

We also put most of what I posted into a series of custom functions so that the global scope isn't getting messy with lots of dataframe objects. Then it's just custom_function(service_layer, hosted_layer).

But again, a lot of those extra details are quite specific to our dataset. The code above should be enough to start the process and modify it as needed.

[Todd_Metzler]

Hello,

You are in luck with one exception.  The exception is download as FGDB option.

workflow:

1.  Maintain your data in RDBMS (SQL server).

2.  Register a read only connection to your SQL server DB with ArcGIS Enterprise.  Suggest a low permissioned (connect, select...) DB authenticated account for this.

3.  Publish a service to your ArcGIS Enterprise as feature service type.  Feature service type data must be hosted in EGDB like SQL server or ArcGIS Data Store.

4.  Add content with the service URL in your ArcGIS On Line Organization.

5.  Share to a group that has open data (aka hub) enabled.  You'll then see the content in open data (aka hub) with the download options except FGDB.

6.  When the content is updated in your SQL DB, the hosted feature layer will see those edits and your end users will too.

EXAMPLE: Hub Content sourced from ArcGIS Enterprise 

Todd

[Kevin_MacLeod]

I see some heavy hitters on this thread. So I want to throw this question out.  Does SDE sync to hosted feature layers out of the box now?  We have 10.9.1, that would be wonderful. But even if it's 11.3 or coming to 11.4 or 11.5 that would be great to hear. 

In a perfect world, hosted feature layers would act just like a referenced service pointing to SDE.  Is that how it works in Enteprise 11.x?  I am guessing no?  I see a script above, to copy data. What if the schema changes?  Also I am wary of python bandaids on the Esri stack unless it's the only way.  Basically, wouldn't it be great if the hosted feature layer was an exact twin of the SDE layer? And the second data changes on SDE, it would push. Or perhaps push on a specified syncing schedule. Same thing with schema, if it changes, sync it, done. I am guessing... no. I'm not sure I see the point of hosted feature layers in this case. Vector tiles are great but it's too much work to leverage.  Frankly I wish Esri had implemented hosted feature layer functionality natively in SDE. My guess, it was too much work to support multiple RDBMSs (SQL Server, Oracle, others).  Which is understandable. But, we need some way of relying on hosted feature layers mirroring SDE.  Possible? Or, is it coming?

And sidenote, hopefully, one day vector tiles will support featureServer cabilities, ie Identify/Query. (basically both things put together in one endpoint, one service sublayer, not two different REST endpoints doubling the amount of layers in an app)

Thank you all, for input or guidance.

I think for now, I'm going to re-architect our current situation, where we have several databases and no versioning..   to Posting all DBs into the Default, and keep that Protected, at the top, so that I can point map viewers to this for read-only services.  Because ..... FeatureServer services require ALL layers to be from one singular database connection.  (I always wondered why that was the case, but at least as of 10.91 that seems to be required still).  One reason is so labels for all layers can be aware of each other in terms of placement. 

[jcarlson]

"Hosted" is shorthand for "stored in the DataStore", which also means "definitely not in the SDE".

The whole point of a hosted layer is to do all the management of schema, access control, etc., through the Portal interface and not bother with administering a database. I'm not sure what a "hosted layer" would even mean if it were just registered to an SDE connection instead.