What would happen to a Layer View if its source is changed by "re-publish"

1331
12
Jump to solution
03-17-2021 01:07 PM
MingHome
New Contributor III

What would happen to a Layer View if its source is changed by "re-publish" from GDB with the same items or deferent number of items?  For example previously existing item may be no longer there and new items may be added. I am wondering, what would happen in this case?

 

Ming

0 Kudos
1 Solution

Accepted Solutions
jcarlson
MVP Notable Contributor

That's correct, Append won't catch deletes. I have a process for that, but it does require that the source dataset has a globalID field and the destination dataset has a GUID field of corresponding values.

  1. Query the values of the the globalID in the source dataset.
  2. Query the destination dataset for all rows where the GUID field is not in the globalID list
  3. Pass the resulting objectIDs to the deletes parameter of edit_features.
feature_layer.edit_features(deletes=[objectids])

I'm not really sure if there's a backwards append from AGOL back to some other layer, at least, not a built-in method.

As you mention, dealing with deletes and other layer types means it may make more sense to just develop your own system, which is in fact what we do for updating some of our layers. We use SEDFs and the merge function a lot. Here's a simplified version:

Depending on your data sources / destinations, what precedes this may change. This bit of code assumes that you already have your two datasets converted into SEDFs.

# Merge in the shared field only
merge_df = src_df.merge(dest_df.loc['shared_field', :], on='shared_field', how='outer', indicator=True)

adds_df = merge_df.loc[merge_df['_merge'] == 'left_only', :]
updts_df = merge_df.loc[merge_df['_merge'] == 'both', :]
deletes = merge_df.loc[merge_df['_merge'] == 'right_only', :]['objectid_field'].to_list()

destination_layer.edit_features(adds=adds_df, updates=updts_df, deletes=deletes)

 

Also, if your destination layer is a FGDB, you may instead use the SEDF's to_featureclass method and overwrite the destination layer. As noted above, this process will depend a lot on your situation.

In ours, we were able to identify a dozen or so consistent, common procedures in our updates, and wrote them into a few custom functions.

- Josh Carlson
Kendall County GIS

View solution in original post

0 Kudos
12 Replies
jcarlson
MVP Notable Contributor

As always, the answer is "it depends".

If you remove an item or a field that is in the layer view, you'll probably break it. If you add new layers and fields, the view may work, but it may not. We use hosted layers with views all over our organization, and we have seen a whole assortment of outcomes when republishing layers with views on them.

If your view layers aren't heavily used, it may be best to just create new views once you republish. We have a few view layers that, while they continue to work after republishing the source data, seem to lose certain functionality.

- Josh Carlson
Kendall County GIS
0 Kudos
MingHome
New Contributor III

Thanks for the sharing!

This is what I am afraid of. It seams that we should keep the scheme of the feature layer service as stable as possible. In stead of "re-publish", use other methods, such as append, to synchronize the source data in gdb and target in AGOL. Any suggestions on the "sync methods"?

0 Kudos
jcarlson
MVP Notable Contributor

If you're using AGOL for the hosted layer, the built in append command (see also here) works very well, and picks up updates as well as adds.

If you have a shared identifier field between the GDB and the hosted layer (like a globalid in the GDB and a corresponding GUID in the hosted layer), there is also a lot you can do with spatially enabled dataframes to identify adds, updates, and deletes between the two sources, but that will depend a lot on your data.

- Josh Carlson
Kendall County GIS
0 Kudos
MingHome
New Contributor III

Thanks for sharing!

I will take a close look of it!

 

Ming

0 Kudos
MingHome
New Contributor III

Hi Josh,

Thanks for your reply! I am developing a workflow to keep a large dataset/feature service in sync between FGDB and AGOL. Your reply and very helpful and give me the direction to start with.  I spent sometime to study it and have two follow-ups questions regarding the "append" method:

1, It seams "Append" can update the feature its source has changed and insert new feature that is not present in target layer. But I did not see that "deletion" function. How do you deal with the feature(s) that exist in the target but not present in the source layer?

2. Append is designed for updating AGOL layer with various sources. Is there something similar for updating FGDB featureClass with a source layer from AGOL. I have some feature layers that is maintained in AGOL with collector and webmap apps. However, I also keep a copy in fgdb and would like to keep them in sync with the "source" in AGOL. 

The advantage of the "append" is that it ability to identify "changes". If I am going to develop a "automated" process to keep the fgdb and agol in sync on both ends, it may be easier just to develop a "change detection" routing and using featureLayer.(add, update, or delete) methods.  In this case, the SEFD may become handy. I am going to take a look of it and like to hear more your opinion and experiences about it!

Again, thanks for sharing!

Ming

 

0 Kudos
jcarlson
MVP Notable Contributor

That's correct, Append won't catch deletes. I have a process for that, but it does require that the source dataset has a globalID field and the destination dataset has a GUID field of corresponding values.

  1. Query the values of the the globalID in the source dataset.
  2. Query the destination dataset for all rows where the GUID field is not in the globalID list
  3. Pass the resulting objectIDs to the deletes parameter of edit_features.
feature_layer.edit_features(deletes=[objectids])

I'm not really sure if there's a backwards append from AGOL back to some other layer, at least, not a built-in method.

As you mention, dealing with deletes and other layer types means it may make more sense to just develop your own system, which is in fact what we do for updating some of our layers. We use SEDFs and the merge function a lot. Here's a simplified version:

Depending on your data sources / destinations, what precedes this may change. This bit of code assumes that you already have your two datasets converted into SEDFs.

# Merge in the shared field only
merge_df = src_df.merge(dest_df.loc['shared_field', :], on='shared_field', how='outer', indicator=True)

adds_df = merge_df.loc[merge_df['_merge'] == 'left_only', :]
updts_df = merge_df.loc[merge_df['_merge'] == 'both', :]
deletes = merge_df.loc[merge_df['_merge'] == 'right_only', :]['objectid_field'].to_list()

destination_layer.edit_features(adds=adds_df, updates=updts_df, deletes=deletes)

 

Also, if your destination layer is a FGDB, you may instead use the SEDF's to_featureclass method and overwrite the destination layer. As noted above, this process will depend a lot on your situation.

In ours, we were able to identify a dozen or so consistent, common procedures in our updates, and wrote them into a few custom functions.

- Josh Carlson
Kendall County GIS
0 Kudos
MingHome
New Contributor III

 

Hi Josh,

For testing, I created a small dataset with a sharedGUID field and published it to AGOL. I successfully created source_sdf (featureClass from fgdb) and dest_sdf (FeaureLayer from agol). However, I encounter a problem with the merge operation (due to miss match of the sharedGUID?). I think that the problem might be the syntax of the GUID. In the featureClass (source), both GlobalID and sharedGUID (coppied from GlobalID) are in upper case and with "{}". However, once it publish to AGOL, both GlobalID and sharedGUID became lower case in FeatureLayer but still shows the "{}". As the results, the sedf created from featureClass has it value of GlabolID and GUID in upper case with "{}", but, the sedf created from featureLayer has its value of GlabolID and GUID in lower case and without "{}".  IT seams that it resulted in miss match. Am I correct about the miss match? If so, how do you resolve this?

 

Thanks a lot!

 

Ming

0 Kudos
jcarlson
MVP Notable Contributor

I haven't encountered that problem... You made sure that the shared field was explicitly a GUID type?

- Josh Carlson
Kendall County GIS
0 Kudos
MingHome
New Contributor III

Yes, this is for sure. It is created in featureclass, value is cppied from GlobalID. Then,  published to agol.

 

Ming

0 Kudos