Community
Select to view content in your preferred language

AGOL Hosted Feature Layer Administrative workflows and Data Maintenance.

959
0
07-06-2021 08:08 PM
Status: Open
by Anonymous User
Not applicable

For AGOL Administrators supply a better-documented workflow for managing and maintaining the data in their organisations.

The Background

As a user of AGOL for a very long time, I have some services that have now been used in a production environment for over 3 years. Now while there are some good administrative workflows for Enterprise layers one area that is significantly lacking is the details about what maintenance tasks are performed by the black box of AGOL. My incorrect assumption was that this was fairly robust but what I have found is that with the offline abilities of AGOL Hosted feature layers there are several scenarios in which the Offline Data Replica can be orphaned (User leaves the organisation, map fails to download in Collector/field maps, uninstall the app, loose device, break the device, etc). Now if you know anything about Replicas the longer they aren't synced the larger the overhead for the service and overall poorer performance. In a recent troubleshooting Ticket for an unrelated matter, I was looking at the replicas for one of my services and found it to have almost 2000 replicas. Hence its crazy size on AGOL despite its actual data only being around 400Mb it was pushing the 10Gb limit for AGOL hosted feature layers not that that limit is documented in the Publish AGOL Hosted feature layer help.

The interim Solution

My Python notebook experience is rather limited I have only just recently Dug into them and am more of a google search and manipulate rather than a programmer. However, I was able to build the two following Notebooks that can be run in AGOL to search a specified layer, and identify Replicas owned by users that are no longer in the organisation (for me reduced replicas by 300ish). 

The second script finds replicas that are over a year since they were last synced.

Now, this isn't a solution but a workaround until this can be resolved by a more elegant solution from ESRI.

Old Replicas Code

# Administrator Script to Look at Orphaned Replicas

## Welcome to your notebook.


#### Run this cell to connect to your GIS and get started:


```python
# Import all the stuff make sure you are loggin as the Data Owner
import arcgis
import time
from arcgis.gis import GIS
gis = GIS("home")
```

#### The Varriables For use in the script


```python
# Item ID of the AGOL Hosted feature layer to Analyse.
item = gis.content.get("xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
# A time the Replicas Last Sync Datetime Stamp will be compared to.
time_epoch = time.time() - 31413600  # Currently Set to 1 year ago.

```

#### The Containers


```python
#Dict to hold information about the Replicas.
REPLICA_DICT = {}
# The List of Replicas that will be deleted.
DEL_REPLICAS = []
```

#### Now you are ready to start!


```python
item
```


```python
# Build the Feature layer collection to find the Replicas on.
item_flc = arcgis.features.FeatureLayerCollection.fromitem(item)
type(item_flc)
```


```python
# Build a list of all of the replicas within the Feature Service.
replica_list = item_flc.replicas.get_list()
```


```python
# Building a Dictionary to hold information about the Replicas we are interested in.
for r in replica_list:
    temp_r = item_flc.replicas.get(r['replicaID'])
    REPLICA_DICT[temp_r['replicaID']] = [temp_r['replicaID'], 
                                         temp_r['replicaOwner'], 
                                         temp_r['creationDate'] , 
                                         temp_r['lastSyncDate']]
print('Built Replica Dict')
    
```


```python
# Iterate through the replicas and find ones that meet the criteria.
for rd in REPLICA_DICT:
    # Get the replica values.
    test_values = REPLICA_DICT[rd]
    # Get the last sync date and convert to ms.
    check = test_values[3]/1000
    # Compare the Last sync date with the time from a year ago.
    if check < time_epoch:
        # If the last sync date is prior to the time epooch mark for deletion.
        print('Need to delete: {check_time} Owned By: {rep_owner} Created On: {creat_date}'.format(check_time=str(time.strftime("%Y/%m/%d, %H:%M:%S", time.localtime(check))),
                                                                                                   rep_owner=test_values[1],
                                                                                                   creat_date=str(time.strftime("%Y/%m/%d, %H:%M:%S", time.localtime(test_values[2]/1000)))))
        # Get the user to verify that the replica should be deleted.
        # This is a tedious but nessasary step as once the replica is Unregisted there is no way to recover it.
        add_del = input('Do you want to delete it? Y/N')
        if add_del == 'Y':
            DEL_REPLICAS.append(rd)
            print('Added {repid} to delete list'.format(repid=rd))
        else:
            print('Skipped')

# Let the user know how many Replicas they are about to destroy.
print('del Replicas: ' + str(len(DEL_REPLICAS)))
    
```

#### This is the Point of no return Once the next module is executed these child replicas will never be able to sync again. But they shouldn't exist!?


```python
# Iterate through the list of Replica ID's that have been manually added to the list for deletion.
for bad in DEL_REPLICAS:
    print('Un Registering: {bad_rep}, {bad_time}'.format(bad_rep=str(REPLICA_DICT[bad]), bad_time=str(str(time.strftime("%Y/%m/%d, %H:%M:%S", time.localtime(REPLICA_DICT[bad][3]/1000))))))
    # Perminately remove the Replicas from the AGOL Hosted Feature layer forever.
    item_flc.replicas.unregister(bad)
    
    
    
```


```python

```

The Old Users Code

## Welcome to your notebook.


#### Run this cell to connect to your GIS and get started:


```python
# Import all the stuff make sure you are loggin as the Data Owner.
import arcgis
import time
from arcgis.gis import GIS
gis = GIS("home")
```

#### The Containers


```python
# Dict to hold information about the Replicas.
REPLICA_DICT = {}
# The List of Replicas that will be deleted.
DEL_REPLICAS = []
# Stage an in-memory list of all users to notify/delete.
USER_LIST = []
```

#### The Varriables


```python
# Iterate through all of the users in the Organisation and append them to the user list.
# Be cautious that this value is more than the number of users in your Organisation.
# If max users is left blank it should return all but only returns the first 100.
for user in gis.users.search(max_users=3000):
    USER_LIST.append(user.username)

print(len(USER_LIST))

```


```python
# Item ID of the AGOL Hosted feature layer to Analyse.
item = gis.content.get("xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
```

#### Now you are ready to start!


```python
item
```


```python
# Build the Feature layer collection to find the Replicas on.
item_flc = arcgis.features.FeatureLayerCollection.fromitem(item)
type(item_flc)
```


```python
# Build a list of all of the replicas within the Feature Service.
replica_list = item_flc.replicas.get_list()
```


```python
# Building a Dictionary to hold information about the Replicas we are interested in.
for r in replica_list:
    temp_r = item_flc.replicas.get(r['replicaID'])
    REPLICA_DICT[temp_r['replicaID']] = [temp_r['replicaID'], 
                                         temp_r['replicaOwner'], 
                                         temp_r['creationDate'] , 
                                         temp_r['lastSyncDate']]
print('Built Replica Dict')
    
```


```python
# Iterate through the replicas and find ones that meet the criteria.
for rd in REPLICA_DICT:
    # Get the replica values.
    test_values = REPLICA_DICT[rd]
    # Get the Replica Owner for the comparison.
    check = test_values[1]
    # Compare the Replica owner with the list of users from the organisation.
    if check not in USER_LIST:
        # If the User is not a member of the organisation mark for deletion.
        print('Need to delete: Replica Owned By: {rep_owner} Created On: {creat_date}'.format(rep_owner=test_values[1],
                                                                                              creat_date=str(time.strftime("%Y/%m/%d, %H:%M:%S", time.localtime(test_values[2]/1000)))))
        # Get the user to verify that the replica should be deleted.
        # This is a tedious but nessasary step as once the replica is Unregisted there is no way to recover it.
        add_del = input('Do you want to delete it? Y/N')
        if add_del == 'Y':
            DEL_REPLICAS.append(rd)
            print('Added {repid} to delete list'.format(repid=rd))
        else:
            print('Skipped')

# Let the user know how many Replicas they are about to destroy.
print('del Replicas: ' + str(len(DEL_REPLICAS)))
    
```

#### This is the Point of no return Once the next module is executed these child replicas will never be able to sync again. But they shouldn't exist!?


```python
for bad in DEL_REPLICAS:
    print('Un Registering: {bad_rep} , Created on: {bad_time}'.format(bad_rep=str(REPLICA_DICT[bad]), 
                                                                      bad_time=str(str(time.strftime("%Y/%m/%d, %H:%M:%S", time.localtime(REPLICA_DICT[bad][3]/1000))))))
    item_flc.replicas.unregister(bad)
    
    
```

#### The End of the Script