Looking for offline data collection advice and best practices

[Henry]

Hi all,

This isn’t a specific troubleshooting question persay, but I did want to send some feelers out and see if anyone else may have advice, know additional resources or workflows that work well for their organization - especially with Field Maps poised to be one of the main field data collection services.

I am the GIS Services Specialist at a natural resources/environmental nonprofit and we mostly do invasive plant species survey and treatment. We work over a large area (~3 counties) and are often working in rural or mountainous areas without any cell service. The data we collect is detailed, down to monitoring over a dozen plant species and recording phenology and substrate location information, in addition to photo attachments which leads to ~4000-7000 new records per year.

Data collection in offline areas is often a hassle. We currently have an Azure VM that hosts an ArcGIS Enterprise Portal, Server, and DataStore, with our field crew syncing their results after returning to good service or wifi. We have used a variety of Esri products for our offline data collection to mixed results:

• Survey123 works well for our detailed form collection, and for point data collection in offline areas, however when collecting polygons in offline areas it has to have a linked map with an offline area component, except the linked map area doesn’t show other feature service layers so it can lead to frustration when returning to a site and being unable to see previously collected data for context.
• Collector/Collector Classic: We used Collector Classic for a long time because it was the most reliable, and it could make use of hosted tracking layers. We have dabbled in using the new Collector but when it came to downloading offline areas with our hosted feature layers it was extremely unreliable. Hosted tracking layers going forward appear to require a GeoEvent Server and Temporal Big Data store which I haven’t gone through and configured yet. Plus, there is a deprecation notice for Collector.
• Field Maps: Still getting used to it, it is similar to Collector which has led to similar problems with downloading offline areas. Offline map areas created in the Web Map through Portal reliably do not download and creating offline map areas from the mobile device works better on occasion but still fails often, even over wifi. What’s frustrating is that the Portal-created offline map areas are usually small subsets of aerial imagery and our hosted data, often no more than 15-20 MB at most, but just yesterday one of our team successfully downloaded a huge mobile-created map area that was 2GB!

I’m working through how I can improve this for everybody on our team and have gotten myself pretty stuck. I’ve gone through the services and web maps dedicated to this offline data collection to confirm that they contain Hosted Feature Services with sync enabled, offline imagery, and am still encountering problems. I’m currently working on coming up with a strategy for sideloading some aerial mosaics – but my tile packages have come out too large for many of our crew’s mobile devices.

I’m wondering if anyone has had any success in data collection using ArcGIS Field Apps in primarily remote/offline areas and would be willing to impart their knowledge/workflow that gave them the best, most reliable results? Is it more advantageous to sideload one giant tile package or make smaller subsets focusing on specific areas? Is it better to include reference data such as roads and rivers in the tile package and keep only a small number of hosted feature service “data” layers for collection - thus reducing how many features are active and query-able? We often need high quality aerial imagery of watersheds and river systems so the imagery gets to be fairly detailed.

Just curious how others’ experiences have been using Esri field apps in offline areas, and what has worked best for them/what hasn’t.

Thanks!

[Linde_Carmack]

I can't comment much on the offline editing component of your question, but downloading maps is an essential part of the user workflow for my organization. Users download multiple maps each, are downloading 200+ square miles of coverage in each offline map, and need detailed imagery to support utility work. Sideloaded TPKs are absolutely critical to support this. We have gone the route of one very large TPK rather than multiple smaller ones, just for ease of management. That TPK is 5-10gb, so device storage is definitely a limitation with that option. All of that said, downloading very large areas even with a sideloaded TPK is not 100% effective. I would say 1 out of every 10 attempted downloads fail, but our users just know to try again and not worry unless it starts happening consistently. We recently made the jump from Classic straight to Field Maps. I think the download success/reliability is probably about the same, but users have found the download process easier and more intuitive in Field Maps. 

Not sure how much of our process is sanctioned Best Practice, but it is what has worked for us. 

[Henry]

Thank you for the reply, Carmack

It's really great to know more how others manage sideloading TPKs and how Collector vs. Field Maps has been working for you. I'll definitely be working on the TPK route for the future.

Much appreciated!

[DougBrowning · Accepted solution]

Is this a government project?  I just ask since I do NR, species stuff with big forms also.

I have 11 states, 3-7,000 long species lists and collect 100s of thousands of records each year.  We are always offline for sure.  It does make it a lot harder to be offline, esp for long periods. 

123 has always been solid.  We just skip basemaps for 123 since we just do point data.  Collector/Field Maps we side load, usually by state or office area. Collector sync is nowhere near as solid as 123. Trick for TPKs is to not go too detailed.  Anything past L15/14 or so you do not see much diff anyway.  My last job we had 100GB+ of offline TPKs.  Just get a sd card and it will fit.  Note we used a custom app for this that was Esri based. Yes you have to double them up now between apps due to security rules which is a pain.  Used to be able to point 123 to Collector basemaps but not sure you still can.  VTPKs can be much smaller but not much help for imagery (but all else they are tiny).  If you go big then side loading is really the only way. 

With areas that big it will always be painful.  Just plain a lot of data.  Have you looked at offline areas?  I never really got it to work but that is the concept.  One map per county could help.

Being able to take large data offline is for sure a pain in the current tools.  Esri tends to assume small projects and it tends to break down at scale.  Things like offline areas require multiple maps to manage it which is tough when you have multiple contractors or areas.

Hard to type all of this but if you are government related feel free to DM me.  Hope that helps.

Oh and yes Tracker will now cost extra and all goes to one big service which is super weird.

Hope that helps.

[Henry]

We're not part of an agency, but we do partner with tribe, county, state and federal agencies in the area and to collaborate on invasive species identification and management. Much of our funding comes from state conservation grants.

123 has been our main tool because of its form customization and reliability so far - although our field crew likes polygons because they can better capture their survey extents and treatment areas - and having a basemap that they can see land ownership and hydro features helps.

Offline areas have not worked well at all, same unreliability with downloading, even for really small subset areas. I thought being able to have a hierarchy of Site Area -> Watershed Web Map for collection would have made things easier but it didn't seem to change much.

Looks like I'll be sideloading moving forward, and with larger areas. Good to know some tricks for the TPKs.

Super helpful to hear how others are handling this process, much appreciated!

[DougBrowning]

I have forms for LPI, Species Richness, Plot layout (Char and Observation), Gap, Soil Stability, etc for Terrestrial.  Also a full suite for Lotic and Lentic sites.  We share with several orgs and partners so if you ever want the forms or methods just let me know.