Field Maps and data transformation

[TanithLives]

Hi all

Requiring some reassurance regarding some workflow I have established for Field Maps data collection. This is building upon lots of stuff I have read on this forum and in the ESRI blogs, but require a hard-and-fast answer.

I need to know if the following process ends up with quality, high-accuracy data, or something that has been transformed into an ugly mess.

1. Field Map set up, creating feature layers based in GDA2020 Zone 56 (an Aussie system), using high accuracy GPS metadata

2. Basemap is ESRI world imagery (for export), making entire Field Map have a CRS of WGS84

3. GNSS set up to collect in GDA2020

4. Field Map location profile is performing a transform GDA2020_To_WGS_1984_2

5. Data collection occurs

I would like to know if I am losing anything in this process, particularly as it reads to me like the data is going GDA2020 > WGS 84 > (then presumably back to) GDA2020 z56. In particular, is the transform happening from the basemap CRS to the required CRS of the feature layers (GDA2020 z56)

Generally I don't have capacity to create my own basemap (therefore avoiding the WGS84) and the feature layers have to be set up in GDA2020 z56.

Anybody got thoughts on this process? Any insight appreciated.

Cheers

[RTPL_AU]

Hi @TanithLives  
A couple of things:

• I haven't checked on Pro 3.4 but some earlier versions had a bug that did weird things to GDA2020 data during the upload/publish process. We'd get odd shifts in the data that are not transformation-based but something else. Gave up chasing it with EAU. They said there was a bug between Pro & AGOL and recommended using transformation WKID 9690 but your mileage may vary.
  Bottom line is to check, never trust, & verify - For data collection layers make sure to have at least one object with a known location in the data so you can check if it lands in the right spot after being published.
  
  To see the transformation chain applied to a layer you have to look at the layer's JSON. (Item details > URL >View > Admin > JSON and then you'll have the forward & reverse transformation details)
  
  Make sure this shows what you think it should show. Because Pro may automatically assign a 'best' transformation when you add layers to a map (to quickly check something) you may find that the AGOL published transformation is not what you expect.
  
  Workaround is to have a Pro Map dedicated to publishing layers and check the coordsys & transforms BEFORE EVERY share/publish process. 
  
  
• What are your accuracy requirements? It might just be worthwhile to use native WGS84 collection and then transform the captured data using a process that you fully control.
  
  
• There are some limitations to transformations in Field Maps while working offline. Make sure that you are on top of it / have checks in place to verify results if there is a chance that other people might use populated coordinate fields etc from data in Field Maps i.e. before you/other GIS person has had a chance to check & verify.
  
  
• Try to have a control point that people can capture at the start/end of every shift. Can be as simple as the gate post at the camp exit. Can be a life saver to see if any settings on GNSS receiver where tweaked due to an app update or config changes made while using gloves. Not that anyone in SEQ would be using gloves at the moment...

[TanithLives]

G'day @RTPL_AU . Thanks for the prompt and informative reply. There's a bit there I will need to unpack. 

- the feature layers for data collection are being generated through the Field Maps app (so, AGOL), which means am bypassing any issues with Pro > AGOL. I do use background data from Pro, but that is reference only.

- will check the JSON info when a layer comes in - it would be good to see what has happened

- yeah, every time I export from Pro I use a dedicated map to ensure I export the right type of settings

- we're not playing for sheep stations, but prefer to stick with the GDA2020 for accuracy. My understanding of WGS84 (perhaps wrongly) is that accuracy is lower. Having said that, do you think a high precision GPS using the GDA2020 system, collecting into a WGS84 map with WGS84 feature layers, would still have the accuracy we're after?

- I did not know about the offline problem. We're often collecting offline. Will have to look into that.

- That's a good idea about verifiable points!

Cheers

[RTPL_AU]

Getting into the weeds but the coordinate system itself doesn't affect accuracy (put simply). The issue is with the transformations used - this is where a few cm or a few m can sneak in.

I'm going to defer to your GNSS manufacturer to set the expectation of what method/settings/datum/etc is the most accurate/consistent/supported. When getting into sub-metre you have to check the full collection to output chain to not introduce uncertainty. 

This often happens when different groups start using their own software to do transformations (e.g. when some use CAD, some QGIS, ArcGIS, MapInfo, etc).

If you are wanting to go sub-metre it would be a good idea to add the full GNSS attribute set to your feature classes (there is tool to do this in Pro). This will grab the number of satellites, accuracy & PDOP etc metadata and also include the lat/long from the GNSS receiver - if in GDA2020 you can use this to check against point location that Field Maps created after it did its thing. The fields can be hidden from users to keep the main data they see 'cleaner'.

Having a control point available will allow you to check if captures are as accurate as you think. Control is everything, never assume.

Depending on the hardware & software, make sure you check the location profile in Field Maps. If I remember correctly by default it expects that the data from the GNSS receiver is in WGS84. 

There should be section on all of this in the Field Maps online help under high accuracy collection or something to that effect.

[TanithLives]

G'day @RTPL_AU . Thanks again for the reply. That all sounds promising. Yeah, the data is being collected using the full GNSS attribute metadata. I hadn't thought to actually use this to compare any transformation issues. That's a good idea. I had indeed buried myself in the available documentation about the issue - but getting a straight answer as to what might be going on was always difficult. The Location Profile is also set up correctly, running from GDA2020 > WGS84.

I think I might be unduly worrying about the data accuracy, is it sounds like the collection is happening as it should. The differentiation between what's happening with the transformation and how accurately the data is being collected is an important distinction I had not cottoned-on to. Cheers

[MarkWILSON_LLS]

gday there @TanithLives , did you ever get any resolution on this process. 

Im staring at the same problem for the first time and trying to work out what datum to create the dataset in and how to load/extract data from AGOL correctly pre and post survey. 

Im using an Arrow100 with CorsNet (gda2020) feed. 

thanks

mw

[TanithLives]

G'day @MarkWILSON_LLS . Errr, not really I am afraid. I've reverted to setting up the maps/Field Maps job in WGS 84. Establishing them in a different CRS feels like you're fighting the ESRI ecosystem - unless you want to set your own basemaps up with the required CRS. Also, it means it limits the number of transformations that occur. In our case it will be GDA2020 (RTK unit) > WGS 84 (Field Maps and Feature Layer), then I can export and transform as required. This removes one step in the transformation process, which was formerly: GDA2020 (RTK unit) > WGS 84 (Field Maps) > GDA2020 (Feature Layer).

Having said that, we are conducting a test of both these setups next week to see if we are getting any differences in survey. This should tell us if the extra transformation is actually causing any issues.