topic Re: Is there a workflow to backup Collector while in offline mode? in ArcGIS Collector Questions

Is there a workflow to backup Collector while in offline mode?

NathanEnge — Tue, 02 Aug 2016 22:53:39 GMT

Hi folks

We have the need to use Collector in offline mode for at least a week during field season. Although our devices are charged regularily and have access to be connected to our mobile workstations, there is still no wifi or cell ranges to sync to AGO.

In order to mitigate the risk of losing collected data on devices due to hardware failure, is there a workflow where the data can be backed up through iTunes or the file system (for android).

Re: Is there a workflow to backup Collector while in offline mode?

zkovacs — Wed, 03 Aug 2016 12:07:48 GMT

Nathan

You can find the collected offline data in the ../ArcGIS_Collector/offline_data/ folder on your Android phone. If you have only one survey downloaded for offline use to the device, it's easy to find the subfolder as there will only be one. Project folders are named with a unique id (not sure, but this might be the replica ID generated by the API) instead of a readable name. When you found the project folder in question, there will be a number of runtime GDB files with the extension of ".geodatabase" (depending on the number of FS used). Simply copy the whole folder to back it up (the process should be similar on iOS). You can turn the runtime GDBs in to file GDBs by using the 'Copy Runtime Geodatabase to File Geodatabase' tool in ArcGIS (at least in 10.3).

(Alternative workflow here.)

I hope this helps.

Zoltan

Re: Is there a workflow to backup Collector while in offline mode?

NathanEnge — Wed, 03 Aug 2016 17:13:44 GMT

Thanks zoltan

do you know if there is a similar workflow in the iOS? iTunes doesn't give access to the file structure, nor does just browsing to the device.

May have to use something like iFile?

Re: Is there a workflow to backup Collector while in offline mode?

zkovacs — Thu, 04 Aug 2016 08:45:30 GMT

Hi Nathan

The workflow is fairly similar, although you need to retrieve the content in a different way, since there are restircitions on iOS devices.

1. Using iTunes:

Click on Apps and browse to the bottom of the screen where there is a list of your installed apps which content you can manage from iTunes. Select 'Collector' to see the documents list and then click on the folder with the AGOL username you want to save/backup. Select 'Save' to copy that folder to your PC.

Depending on the size of the folder (attachments, etc), it might take a while to copy.

2. Use a 3rd party app like iExplorer, iFunBox, etc to get access to the file system and only grab the folder/data you need.

Hope this helps.

Re: Is there a workflow to backup Collector while in offline mode?

JosephMcGehan — Mon, 11 Nov 2019 06:14:30 GMT

.. I'd like to use py to automate the above. I'd like to back up all runtime geodatabase, then programaticllay replicate changes against a local master, and pass the runtime geodatabase back.

Re: Is there a workflow to backup Collector while in offline mode?

zkovacs — Tue, 07 Jan 2020 14:49:13 GMT

Hi Joseph,

Not sure how you would access iOS content using Python to be honest. I did a google search but couldn't find any Python module that would allow browsing content on iOS devices. This may have to do with the way iOS is architected: you don't have access to the entire file system by default. It is called sandboxing and is for security reasons: no app (or its data or files) can directly communicate with another app (with some exceptions of course) and use information/data from them.

Most 3rd party desktop apps that allow browsing iOS content rely on iTunes to do this. (iTunes includes a type library that you could use to access its object model after you imported it into your C++ project for example.) I think your first hurdle wold be actually accessing folders/files on iOS devices. After that, there would be the issue of identifying the correct local Collector folder you need for each project - they seem to be named with a unique id (starting with the AGOL username, followed by possibly the replica ID generated by the API) instead of a readable name that may make it hard to programme...

Anyway, if you find a way to do it in Python it would be great to see how you achieved it. I hope this helps.

Zoltan

Re: Is there a workflow to backup Collector while in offline mode?

DougBrowning — Tue, 07 Jan 2020 17:09:02 GMT

This would be connecting to a computer though.

Has anyone done this totally offline? My idea is to use a USB stick plugged into the tablet. I am ordering some to test but have not got them yet.

thanks

Re: Is there a workflow to backup Collector while in offline mode?

DougBrowning — Tue, 07 Jan 2020 19:46:53 GMT

I got this to work in Android copying TO the device... until v6 or v7 came out. Google now names the sd card a random number so you cannot program it.

Re: Is there a workflow to backup Collector while in offline mode?

zkovacs — Fri, 10 Jan 2020 13:31:13 GMT

Yes, it's via a PC. I haven't tried a fully offline workflow as I never had the need. It should be OK on Android as you can copy files to/from SD card and/or USB sticks easily - internal naming is a different matter. iOS doesn't support OTG USB - or at least not without special cables/USB sticks. Then there is the issue of browsing and copying content to the USB stick due to sandboxing (you cannot access the full file system). I hope this helps.

Re: Is there a workflow to backup Collector while in offline mode?

DougBrowning — Tue, 28 Jan 2020 23:52:53 GMT

I got a USB stick for the iPad today. The Sandisk software that came with it will only backups photos.

I then tried Files app and here is where is gets weird. I can see the Collector folders and back them up. But for some reason I do not see 123 folders at all. I know some of this is iOS blocking the file system but if Collector was able to do it then it can be done.

Other idea is - in the Settings -> Storage add a button "Backup database to another location". How cool would that be! Should be pretty easy also. Just click Backup then pick a spot. Could even point it to a cloud service. Please consider it.

We go on 8 day long outings at times. If on day 8 we break a tablet we lose ALL of the data. That means we lose lots of money. Is there any plan for this at all? So far I we tell the client - there is no way to backup in the field, and there is no way to back up AGOL (which is weird cause this could be a big money maker for Esri), and all our master forms are in a location that is not backed up by IT. This is a ton of risk at every level and is not going over well.

Any ideas 123 team?

Next try is Android.

thanks!

Re: Is there a workflow to backup Collector while in offline mode?

zkovacs — Wed, 29 Jan 2020 12:54:00 GMT

Survey123 is slightly off topic here, but here goes as this partially applies to Collector too...

You can publish your feature services (for Collector) or Survey123 surveys to Portal if you have one. That should have backup options set up by your IT/GIS Admin if you're using SDE to store your data. If you're using AGOL or hosting your data in Portal (w/o SDE) to store your data, you can export your data any time you want by going to the feature service's item page and click on the Export button. This can be automated via a Python script using the ArcGIS API (so you can log in to AGOL/Portal, get your FS, create a replica, download it and then delete the replica from AGOL/Portal so it won't take up space) and the script could be set up to run as a scheduled task - every day/week/month/etc.

Your Survey123 forms are always uploaded & saved to AGOL/Portal and you should be able to download them any time you want.

This obviously doesn't solve your issue of offline backups from your devices, but the idea behind these apps was that at some point you connect to WiFi/mobile internet and sync your edits back (to AGOL or Portal). Your best approach to get your ideas across to the dev team is if you log them on the ideas website and get people to promote them. 😉

If your master forms and other data are crucial to operations (and why they wouldn't be), your organisation needs to spend some time and money to implement backups for those.

I hope this helps.

Re: Is there a workflow to backup Collector while in offline mode?

DougBrowning — Wed, 29 Jan 2020 14:44:16 GMT

Sorry I thought I was in the 123 forum as I usually am.

I do have a script I wrote for backup form AGOL (we cannot use portal). It is a full download so towards the end of the field season it takes all nights to do 3 states. We are moving to 6 next year and 11 the year after that so at that point this will no longer be feasible. I have 30-40 Hosted feature services so a manual download is not an option. Out biggest issue here is I must have my computer running and logged in in order to do the download. If I am out or on vacation it is an issue. I am working on getting a VM. Even losing just one crews data one time is 50K. It would for sure be easier to just check a box in AGOL for backups. In order for AGOL to be true enterprise scale it needs to have a way to backup.

I am actually seeing the offline ...database files now in the files app in the iPad. Have not tried a restore but it looks like I can at least see it. I will have to ask the 123 team if they can open up this access as well.

Thanks for the help.

Re: Is there a workflow to backup Collector while in offline mode?

DougBrowning — Wed, 05 Feb 2020 16:43:14 GMT

Update: I have been able to backup to a USB stick plugged into the iPad while offline. I then copied to a PC and used the Copy Runtime tool and it all worked perfectly.

So for the OP Nathan Enge you can backup to a USB stick hanging off the iPad and recover it as needed. No iTunes or PC needed.

I used an iXpand from SanDisk. Only weird thing is you must install the app for it to show up in Files app.

I am now asking the 123 team to please open the folder up like Collector does.

Big thanks to the Collector team for having this open! Good job!