Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

[LanceCole]

We have been testing a high accuracy GNSS GPS with Collector (Aurora and 18.0.1) in a development environment and utilizing ArcGIS online as our Portal.  As we plan to move to the production environment questions have arose pertaining to minimizing data translations to maintain accuracy.  The GPS utilizes and outputs in  the GCS NAD 1983 2011 coordinate system from our State VRS network.  We have been translating this to WGS 1984 Web Mercator Auxiliary Sphere for use with ArcGIS Online and its related basemaps.  Our production system also utilizes the GCS NAD 1983 2011 coordinate system, specifically the NAD 1983(2011) State Plane and no translations are needed between the GPS and production servers; however, we plan on continuing to utilize ArcGIS online as our Portal for collector.  What is the best or recommend approach to maintain the collected coordinate data accuracy?

We are considering:

1) generating our own basemaps on our local servers using the GCS NAD 1983 2011 coordinate system and pushing these up to ArcGIS online and sideloading them directly onto our Collector systems (when/if they will fit).  This would save on credits used for building the tiles but we will still be utilizing credits for storage.  Subsequently, we would publish our features utilizing the same coordinate system. This is currently our preferred approach as there would not be any translations required.

2) continue with the current approach of collecting GPS data using GCS NAD 1983 2011, translating to WGS 1984 Web Mercator Auxiliary Sphere on ArcGIS Online and then translating back to the NAD 1983(2011) State Plane we utilize internally.  We have noted a definite shift in the data collected utilizing this method when compared to data collected by our surveyor utilizing a total station or survey grade GPS on our own local network.  I have experimented with different translations but the WGS_1984_(ITRF08) to NAD_1983_2011 we are utilizing provides the best match.  

3) a question for the ESRI team is it possible to utilize the ArcGIS online basemaps without Collector automatically using the WGS 1984 Web Mercator.  We did try and publish a feature using the GCS NAD 1983 2011 coordinate system but the Maps/Collector automatically adopts the basemap coordinate system.  We were also not positive on how the point data was being handled with a mixed coordinate system and most likely introducing further translations and potential errors.  

It is ironic that a few years ago GIS data that was accurate to a couple of meters we considered fantastic and now we are looking at a target accuracy of 10cm and achieving an actual accuracy of less then 2cm

[AlinaTaus]

Hi Lance,

First, I wanted to thank you for taking the time and sharing your knowledge on this topic. I too have been struggling for a few months now to get our set up right, so it's somewhat comforting to know I'm not alone  

We bought the EOS Arrow Gold units his last fall and had the setup you've described in the other post where I had to run postprocessing to get the Z elevation. Within the last few months EOS came out with support for GEOID12B and Collector Aurora now supports z-enabled features so I had to redesign my workflow. We also updated our orthoimagery and now the basemap we are using is in a different coordinate system than the rest of my GIS data. I am wondering if you can offer some advice on my setup. Here is what I have: 

• The basemap is a hosted service by the State of Maine and it's in GCS NAD 1983 2011. It is also projected to NAD 1983 2011 UTM Zone 19N (meters)
• We will be using the State's RTK, which outputs data in GCS NAD 1983 2011. 
• Our existing GIS data is in NAD 1983 StatePlane Maine West FIPS 1802 (feet)

The first question I have is, when setting up my z-enabled feature class for data collection, I have to set both a horizontal and the vertical coordinate system.  I know my horizontal has to match my basemap, but it is the geographic coordinate system or the projected one? Second, for the vertical coordinate system I have to select what the EOS receiver will output, so this will be NAVD 1988 (meters). Now, for the Location Profile in Collector, I would think that I wouldn't need to set one up, since both the GNSS receiver and the basemap use the GCS NAD 1983 2011. If this is correct, then the only post-processing I would have to do is to reproject the feature class to the State Plane coordinate system and change the vertical to NAVD 1988 (feet) to convert my elevation from meters to feet. Does this sound right? Am I missing a step? 

Thank you very much!

Alina 

[LanceCole]

Alina,

I am not going to be able to help much.  I have not had the opportunity to try the new z-enabled capability of Collector for ArcGIS (v19.0.1).  This is primarily supported with the Arrow/EOS receivers and software and we are using Leica receivers that, unfortunately, have not caught up yet.  Your workflow outline appears to be sound based upon my understanding.

The main issue with data obtained from Collector is "What coordinate system is my data in?"  Going from a GPS receiver, running on various RTK systems, on different coordinate systems, that each manufacture integrates differently with Collector, pushing to AGOL, that uses base maps in one coordinate system but allows you to publish your features in another, back to ArcGIS Pro that does not clearly define which coordinate system it is currently working in, and the fact that there are hundreds of possible combinations of coordinate transformations, all to have your data not appear where you think it should be appearing, or is the data right and the map/image wrong.  No wonder it is confusing.

One thing I can highly recommend is to create a set of control points.  I have two sets I use, one is a small number of points around the office neighborhood that I can quickly go for a walk, on a nice spring day, and test a new collector map.  The second is about 24 points over a large area that I can use to test at finer level that can catch things such as datum shifts that may not be visible on a small localized set of points. We collected these control points using a survey grade GPS, using the coordinate system we ultimately will be working with and typically using 5-10 minute averaging at each point.  The points have been collected at several different times and the results averaged.  I keep them in a shapefile or GDB feature class that is readily accessible.

When I am working with a new Collector Map or trying to figure out is my new transformation workflow model working correctly, I go for a walk and collect these points.  Once collected I bring only the points into an ArcGIS session, perform my transformation workflow and then bring in my controls points to see how they compare.  Sometimes my collected data shows up in the middle of the Indian Ocean, other times it is a few feet off but most of the time it is within the accuracy of the GPS.  When it does not work I tweak my workflow and run it again, check my map or GPS configuration, recollect the points and check it again.  It is simple and quick and I am checking against a single absolute known.

Working with multiple coordinate systems in you data is difficult.  We are in the process of migrating from NAD 83(86) to NAD 83(2011) the difference between the two systems can be 2+ feet.  The key here is to make sure you have the correct transformation assigned in ArcGIS between the two systems.  There is also a performance price to pay for working with multiple coordinate systems as some of your data needs to be projected on the fly.  This is most notable when it is your imagery that is being corrected. The best option is to work in a single system or as a second option work in your imagery coordinate system to minimize processing time.  Another option, if possible is to obtain a physical copy of the imagery, is to reproject the imagery to your preferred datum.  Just plan on some serious processing time depending upon the extent and image quality.

Sorry I cannot specifically answer your question.  I hope this helps.

[AlinaTaus]

Lance,

Thank you for responding. I had plans to go out this week and find some of the survey benchmarks located around town and try to collect their location using my setup but we got 4" of fresh snow dumped on Monday, so will have to wait a bit... Ahhh winter!  Anyway, another issue I discovered yesterday was that the web layer you publish to ArcGIS Online will be in the coordinate system of your data frame, so you have to make sure that the spatial reference of your data frame matches the one of the feature class you are trying to publish. I noticed this when I tried to bring the web layer back in Pro to look at the data I had just collected and saw that the spatial reference was in my local State Plane and not UTM Zone 19N. However, the data I collected was remarkably close (both XY  and Z) to the data I had collected previously with a Trimble unit. I can't wrap my head around why I was able to get good data (I think!) with a wrong setup, but it just makes me even more nervous and less confident that I can do this on my own.

I do have a copy of the recent imagery and my original intention was to reproject it and cache it on our server (like we've done with some older imagery) but I ran into all kinds of error messages and had both the clip and reproject geoprocessing tools spinning for hours with no progress so finally I just had to give up and move on. I might go back to it at some point...

Anyway, I'm going to try and fix the coordinate system, go out and collect some benchmarks and hope that I'll get some accurate data! Thanks again for your input! 

Alina 

[MitchJohnson]

Hi Lance,

I believe what you have described here is the exact issue i have been facing-- the primary problem being understanding datum shifts and ESRI's projecting on the fly.  I have two questions that maybe you can shed some light on:

1. Testing against NGS Survey Control Monuments.  This is where I first noticed accuracy issues.  When comparing your collected data, would you compare against the GIS files for these NGS Survey control points as they are, or would you re-project them to the coordinate system that your data is in (not your collector basemap projection, but the coordinate system that your feature classes are in).  I want to trust the Survey monument GIS, but does it need any re-projecting if it is in a different projection than my data?

I am using a real-time differential correction service and am believe i know which coordinate system the map and the GNSS should be in, respectively, when setting up a profile.  But what about projection changes comparing the data to control?

2. You mentioned that there are hundreds of potential datum transformations.  Is the only way to discover the correct transformation to test each of them in the field?

Thanks,

Mitch

[LanceCole]

Oh the joys of data transformations.

1) you needed to lookup the data sheets for NGS Survey Control Monument you are using.  These will list the coordinate system and datum used for the published data.  They may be different depending when the monument was set and/or last verified.

2) make sure you are collecting the correct data using your equipment.  I am fortunate in that I have access to in-house survey control data specific for my location in a known coordinate system and datum.  Check with local surveyors, county map office for survey pin data on plats, state monument boxes for roadways, etc.  You only need a few points to cross check against.  

3) selecting a transformation - I keep our collected data in the datum we work with for our GNSS, in our case NAD83 (2011).  I transform the control data to this system for verification.  This can be difficult but typically common sense prevails.  You know what system your data is utilizing and what system your control was collected with, what is the most logical path between those two. I prefer ArcGIS Desktop for the aspect of determining as you can load your data in and set the data frame coordinate system along with transformations for other systems. The transformations can readily be changed in Desktop whereas I have not found this functionality in Pro. 

Once you have your data loaded in a MXD add your control data and play with different real-time transformation to see what works best.  In our case our control data is in NAD83(1987) but our collector works with NAD83(2011).  We found a combo transformation of NAD83(1987) to NAD83(HARN) to NAD83(2011) worked the best.

4) signs of a transformation issue

    a) all your data is shifted the same distance and direction, within the accuracy of the data

    b) a gradational shift such as data to the north is shifted NE, central data appears to be accurate and southern data appears to sifted to the SW.  Another example is data to the north appears accurate but as you look at data further to the south a larger shift is noted in some direction and increases the further south you go.

    c) some pattern in the data differences

5) accuracy issues - if the differences between your data and the control points are random most likely you are not looking at a data shift but rather an accuracy issue.  You need to know the limits of your equipment and the control data.  Keep in mind if your equipment is rated accurate to 6 or 12 inches that is typically a plus or minus.  You need to double those values and not be looking for cm accuracy.

[MitchJohnson]

Thanks, Lance.

On your points:

1)  Is it better to trust the lat\long from the datasheet and pick my own projection for bring in to GIS or use the shapefile download from the NGS.NOAA website?  There's about a 1.5 foot difference between specifying a North American 1983 HARN datum myself, versus using pre-existing shapefile.

2)  I assume you mean use some proper control points?  I have found a few NGS monuments in the area that our surveyor has said are reliable for testing.

3) Because I am using Collector, the location profile requires a selected transformation between the map coordinate system and the GNSS coordinate system before any data can be collected at all.  Identifying the transformation is the core of this problem.  If i select additional transformations in ArcMap to the already collected data (for which a transformation was specified in Collector), wouldn't I be further abstracting the accuracy by "double-transforming"?

4) I am absolutely facing a transformation issue since point a) that you made is exactly what I've been seeing.

[LanceCole]

1) can you please provide the link to the datasheet you are referencing.  I would like to use something you are actually utilizing to expand my reply.

2) if your surveyor has a survey grade GPS rover pole, have them shoot a series of points over an area that is readily  accessible.   I like the center of manhole covers because they are easy to find, big, do not move and can often be seen in imagery.  Shoot how ever many locations you would like to use as check points using averaging for a few minutes at each location.  Repeat on another day and average the results.  You now have a known set of points, in a known coordinate system that you can check your data transformations against.

3) the GNSS metadata that is stored with your collector data contains the latitude, longitude and altitude (height above the ellipsoid) in the coordinate system your GPS is running.  I use this data and project to coordinate system were are using.  I am really looking forward to Collector support for 3D points using our GPS.  

[MitchJohnson]

1) 

Emil GPS

PID: DG4231

Datasheet URL: https://www.ngs.noaa.gov/cgi-bin/ds_mark.prl?PidBox=DG4231

Gene 2

PID : OM1237

Datasheet URL: https://www.ngs.noaa.gov/cgi-bin/ds_mark.prl?PidBox=OM1237

Edina Taylor GPS

PID: DG4236

Datasheet URL: https://www.ngs.noaa.gov/cgi-bin/ds_mark.prl?PidBox=DG4236

2) I can shoot manhole covers, but how do I control for potential errors/shifts in the imagery?

3)OK, that is helpful.  So, in my case it is storing the data in the GNSS coordinate system.

Thanks very much for helping with this, Lance!

-Mitch

[EmilySchad]

Hi LanceCole,

I've published our own aerial (basemap) in NAD 83 HARN State Plane Virginia North FIPS 4501 US Ft with a feature service in same system to be able to obtain GNSS GPS accuracy with our receiver (arrow Gold) but even though I set up my local profile with Receiver GCS NAD 83 2011 and map NAD 83 HARN ST VA North US Ft the collector will not load my map. It says map coordinate system does not match local profile. Is this because the collector is automatically assigning the map WGS 1984 Web Mercator? If so, is this what ESRI was planning on fixing?

Thanks for your input. 

Emily

[JenniferStarbuck1]

I have all ways see this as an issue from the OG Collector app and still in Field Map app. I just got done doing a GPS Accuracy test using R2 with RTX, R1 with SBAS connected to iPhone8+ and Trimble TDC600. I took the steps to save my data projection in NAD832011, set trimble mobile map (TMM) to get the NAD83 output, configured Field Map (FM) with the location profile datum transformation NAD83 to WGS84. All points collected showed a 1.5ft shift when comparing GPS collected point against photo identifiable targets and survey control points. I performed a second round of GPS collection with TMM outputting WGS84 and FM location profile set to default. This setting gave me the results I was looking for. I do not believe any of Esri field mapping applications can WGS84.