Using a python notebook to append features to a different layer for "storage"/backup

[sophered]

Hello,

I am currently trying to use python to essentially backup features from a "Tracking" layer to a hosted feature layer (I'm gonna call this layer the "repository layer") for extra storage. Originally I was using a notebook to create a spatial data frame of the tracking layer, compare it against the global ids in the repository layer, and append those features if those global ids aren't in there.

My current issue is that as of a few months ago, my script stopped working. After some troubleshooting I realized that the issue lies in trying to load my entire layer into a spatial dataframe. This worked previously, but now it only lets me load up to 180 features, if I try any more than that, it gives me a JSON Decode Error. 

Here is what my troubleshooting currently looks like. I'm wondering if there is a way to get around this, if i should avoid dataframes entirely and if I should try an entirely different approach. I was testing around with Returning IDs Only in my query and trying to use those ids to have a similar functionality, but I don't even know where to start with that. Any help would be GREATLY appreciated! 

[BobBooth1]

If your tracking layer has dates or date/times on the samples, you could use those to select the features created since last backup. You can create a text file in your notebook's files folder and record the last backup date there when it runs.

Another way to handle it is to add a "processed" or "Done" field to your tracking layer, with a default value (I do this with Survey123 feature services). I use integers, to make the expression simple, and calculate their new value to some different value after they're processed up (say 0 and 1). Then each time the script runs, select the features that have the default value and copy them to the back up layer.

example:

# Get the survey results layer and call the FeatureLayer
item = gis.content.get("2522222e41111111116f75448222")
surveyResults = item.layers[0]


FS = surveyResults.query(where='Done = 0')
print("Number of new surveys: " + str(len(FS)))
newResponses = FS.features
#
# do something with the features
#
#
# Now that the new records have been processed, update the Done field so they won't be read next time.

for surveyResponse in newResponses:
    AGOedit = surveyResponse
    AGOedit.attributes['Done'] = '1'
    updateResult = surveyResults.edit_features(updates=[AGOedit])