How do you back up content in ArcGIS Online?

HeathBrackett1 · ‎09-19-2014

Does anybody have any recommendations for backing up Feature Services in ArcGIS Online? I would ideally like to be able to save the Service Definition files to one of our servers and use them to recover lost/deleted data. The problem is that the Service Definition file is only relevant to the time that the Feature Service was created. So if I change the popup configuration, change symbology, or edit features within ArcGIS Online and "Save Layer", these changes are not updated within the Service Definition file. We need a way to create a recoverable snapshot of our data.

Is there another way to handle this? I believe that the popup configuration and symbology can be accessed through the JSON, but you can't access feature geometry and table records through JSON. Is that correct?

Does ESRI have any plans for lost data recovery in ArcGIS Online?

Thanks,

Heath

CamBarnard · ‎12-02-2014

Hi Heath,

It may help to clarify your intent.

Is your concern 'macro' in terms of catastrophic infrastructure failure? ... ArcGIS Online should be taking care of that level of redundancy and protection without conscious need for you to think about it.

Is your concern 'macro' in terms of someone accidentally deleting the entire feature service? ... if so then cloning the service into your private area at some interval might do the trick.

Is your concern 'micro' in terms of how to undo a specific erroneous edit on the part of a particular user?

Other may be able to provide a more specific response if you nudge us in the right direction.

-Cam

RebeccaStrauch__GISP · ‎12-02-2014

Also re: Cam's response "Is your concern 'macro' in terms of someone accidentally deleting the entire feature service? ... if so then cloning the service into your private area at some interval might do the trick."

For each item, you can enable "Delete Protection" by editing the properties. This won't help with something being changed (or corrupt), but it will prevent it from being accidentally deleted.

HeathBrackett1 · ‎12-03-2014

Thanks for the suggestions. I think this is a pretty relevant topic because of the shift in software infrastructure. I'm not necessarily concerned with the total system wipeout. That is definitely up to ESRI to handle. And I'm not particularly concerned with errors in feature editing, either. (Although in an enterprise system we have ways to restore entire systems AND individual files if ever needed.) And I haven't yet addressed the issue of possible file corruption in ArcGIS Online where you are forced to delete and recreate a (hosted) feature service, but it's something to consider.

My primary concern at the moment comes as a result of my organization's switch to using enterprise logins, which requires migrating accounts. The content of each account needs to be transferred to a newly created account and I have written scripts to handle this. These scripts access Python modules that ESRI has created. I'm trying to account for all potential failures. What happens if an internet connection is lost while content is being moved from one user to another? What if there's a power failure during account migration? I don't appreciate that I have to just cross my fingers and hope that nothing goes wrong.

Here are a few of my observations about (hosted) feature services in particular that concern me.

A service definition cannot be updated in ArcGIS Online. Meaning, any reconfiguring of popups, editing of features, or changing symbology/transparency/scale dependency in a feature service while in ArcGIS Online is not reflected in the service definition which essentially makes the service definition useless if anything in the service is changed.
Web maps reference feature services based on the Item ID, a unique ID assigned by ESRI. So if a feature service is lost and needs to be replaced, all web maps that reference that feature service must also be updated regardless of whether the name changes or not. I'd like to have a way to restore that feature service and maintain the original Item ID so that it does not break the web maps that consume the service.
I see that you can download and store all of the descriptive properties of a (hosted) feature service as a json file via the REST API, but I don't know of a way to access the geometry so that both could be combined to restore an individual feature service. (I have not tried to recreate an item based on the stored json file to see if I could even recreate the item with the same original Item ID).

I'm very curious to know if any other organizations are having these same conversations internally and how they are handling these situations. I'm also curious about whether ESRI has any plans to introduce backup capabilities that also restore the original Item ID. If we are to truly dive into ArcGIS Online, we have to have some assurances that we have the ability to restore data under any circumstance.

Heath

BradySmith · ‎12-05-2014

Heath,

It is possible to download an entire feature service in file geodatabase format that is hosted in ArcGIS online. There might be a easier process but one way we have accomplished pulling out the geometry of a feature service is using the export to GEODB and then downloading the database. If it does not show up check your edit settings on the feature service. We have used this process to make schema changes to the db and then publish a new service.

HeathBrackett1 · ‎12-05-2014

Thanks for the response, Brady.

I am aware that you can download the geometry and schema in gdb format. But that doesn't include symbology and popup configuration, nor the item description. I don't want to republish a service, I want to replicate it exactly as it existed.

I'm pretty sure that you can get all of the symbology, popup information (and the original Item ID) in json format, but I still am not sure how to put those pieces back together to exactly recreate the item that these pieces are backing up.

Is there a way to create an empty item in AGO from the downloaded json file that includes the original Item ID, and then load the schema and geometry tables into that item so that it does not break any connections in maps and apps that reference the item?

HeathBrackett1 · ‎12-03-2014

Thinking more about the migration process in implementing enterprise logins, I'm less concerned about losing content through script failure. The process involved in moving content from one user to another involves just changing the value of 'owner' for each of those items and does not involve any copy/delete process. So I assume(?) that if a script were to fail during the migration process, the worst that would happen would be that the content would be split between the two users - which would require manual movement or creation of another clean-up script.

My observations/concerns that I mentioned above still stand, though, and I'd be very interested in hearing ideas on how to back-up and recreate content with the exact specifications of the original content.

Cheers,

Heath

Anonymous User · ‎01-18-2016

Yes, wouldn't that be nice?!!

I have to manually recreate my web maps and web apps whenever I recreate my service. Right now, due to a known bug that doesn't read the changes to my existing service. What a pain.

Going by my human memory and hand-written notes on how I have the web maps and apps set-up.

And if I have 50 hosted feature services, why in the world do I have to go to each individual listing in My Content and download as a fgb? Would be nice to have a "back-up" or "download all" button.

Sorry....mostly a ranting post. But I wanted to agree with what you were looking for.

EDIT: found Backup & Restore web map and web app configurations in ArcGIS Online or Portal

might be helpful

Thanks,

Anonymous User · ‎01-19-2016

Just posted this to Ideas site. Vote it up and we'll see if anything happens.

http://ideas.arcgis.com/ideaView?id=087E0000000kBGR#sthash.g2f4xf9O

NorthSouthGIS · ‎10-18-2016

FYI: You can use the ArcGIS Online Assistant to move an entire content structure from one place to another, whilst maintaining all you ask for. I only wish I saw your question sooner to show you this. It has some limitations, but I believe it solves your problem. You can also log into Pro from multiple Orgs and Portals now and use data from multiple sources to build maps and apps or "Projects". I wonder if there is a way to leverage Pro to achieve this process as well...