topic Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice in ArcGIS Collector Questions

Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

LanceCole — Wed, 23 May 2018 02:47:19 GMT

We have been testing a high accuracy GNSS GPS with Collector (Aurora and 18.0.1) in a development environment and utilizing ArcGIS online as our Portal. As we plan to move to the production environment questions have arose pertaining to minimizing data translations to maintain accuracy. The GPS utilizes and outputs in the GCS NAD 1983 2011 coordinate system from our State VRS network. We have been translating this to WGS 1984 Web Mercator Auxiliary Sphere for use with ArcGIS Online and its related basemaps. Our production system also utilizes the GCS NAD 1983 2011 coordinate system, specifically the NAD 1983(2011) State Plane and no translations are needed between the GPS and production servers; however, we plan on continuing to utilize ArcGIS online as our Portal for collector. What is the best or recommend approach to maintain the collected coordinate data accuracy?

We are considering:

1) generating our own basemaps on our local servers using the GCS NAD 1983 2011 coordinate system and pushing these up to ArcGIS online and sideloading them directly onto our Collector systems (when/if they will fit). This would save on credits used for building the tiles but we will still be utilizing credits for storage. Subsequently, we would publish our features utilizing the same coordinate system. This is currently our preferred approach as there would not be any translations required.

2) continue with the current approach of collecting GPS data using GCS NAD 1983 2011, translating to WGS 1984 Web Mercator Auxiliary Sphere on ArcGIS Online and then translating back to the NAD 1983(2011) State Plane we utilize internally. We have noted a definite shift in the data collected utilizing this method when compared to data collected by our surveyor utilizing a total station or survey grade GPS on our own local network. I have experimented with different translations but the WGS_1984_(ITRF08) to NAD_1983_2011 we are utilizing provides the best match.

3) a question for the ESRI team is it possible to utilize the ArcGIS online basemaps without Collector automatically using the WGS 1984 Web Mercator. We did try and publish a feature using the GCS NAD 1983 2011 coordinate system but the Maps/Collector automatically adopts the basemap coordinate system. We were also not positive on how the point data was being handled with a mixed coordinate system and most likely introducing further translations and potential errors.

It is ironic that a few years ago GIS data that was accurate to a couple of meters we considered fantastic and now we are looking at a target accuracy of 10cm and achieving an actual accuracy of less then 2cm

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

LanceCole — Thu, 24 Jan 2019 12:46:39 GMT

Follow-up

1) we did end up creating our own base maps and side loaded them onto our collector devices. Not only did this save on credit usage for our ArcGIS online storage but the side loaded maps are substantially faster especially in areas with poor cellular connectivity.

2 & 3) collecor still does not fullly support x,y,z transformations so we have remained with the native coordinate system for collector. This is being worked on by ESRI and hopefully will be available in the near future. We also were apple to confirm the latitude, longitude and altitude attribute values stored when using a GNSS GPS are in the raw data coordinate system for the GPS utilized. Do note the altitude values are currently height above ellipseoid and need to be transform to actual elevation.

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

DonRodgers — Tue, 02 Apr 2019 18:36:18 GMT

Lance, we are looking for a way to collect z coordinate data with our Trimble R2 and the ArcGIS Collector app. Can you tell me how you make the HAE to actual elevation transformation?

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

LanceCole — Tue, 02 Apr 2019 19:41:24 GMT

I would be glad to help. A few quick questions.

1) do you have access to ArcGIS Pro? It is not needed but the approach is different between ArcGIS and ArcGIS Pro.

2) what corrodinate system is your Trimble R2 configured?

3) what corrodinate systems do you want your final data to be in?

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

DonRodgers — Tue, 02 Apr 2019 19:59:04 GMT

Lance: I have been using ArcGIS Pro to make and share the web maps on AGO.

I don't completely understand your question #2. In my profile on my Collector app my GNSS Coordinate System is set to GCS NAD 1983 2011. Is that what you are asking?

I want to end up with horizontal data in NAD 83 and vertical data in NAVD 88.

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

BillFox — Tue, 02 Apr 2019 20:16:44 GMT

Is EOS an option for you?

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

LanceCole — Tue, 02 Apr 2019 20:41:20 GMT

Excellent, I will write up a quick "How-To" this evening using your information. I do have a few more questions

When you say NAD 83, I assume GCS_North_America_1983, WKID: 4269?

On the vertical data do you want the NAVD 88 to be in Feet, US FT or Meters?

I am also assuming you have configured your database and are receiving the GPS metadata with each point when collected to AGOL, correct?

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

BillFox — Tue, 02 Apr 2019 21:13:49 GMT

How to Use Real-Time Geoid Height with Arrow and Esri's Collector for ArcGIS | Eos Positioning Systems

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

DonRodgers — Wed, 03 Apr 2019 14:47:49 GMT

Lance, thank you for your quick and thorough response. I can clearly see that your knowledge base and experience far exceeds mine, but I will try to keep up.

In response to your follow-up questions:

When I say NAD 83 I mean that the Seiler Rep who walked me through the Trimble R2/ArcGIS Collector setup had me enter NAD 83 2011 as the GNSS Coordinate system and as the "to" system in my Datum Transformation box in my Profile. All the data that our organization stores on our in-house databases is NAD_1983_StatePlane_Indiana_East_FIPS_1301_Feet.

On the vertical data I ultimately want my data to be in US Feet.

With respect to metadata fields what I have done is to add metadata fields to a published Feature Service Layer and then added that new Feature Service layer to my Web Map. I have attempted to Configure a feature service to store GPS metadata. The obstacle that I encountered is that the instructions indicate that I should add field names to my feature attribute table and these field names are longer than that field will accept.

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

LanceCole — Wed, 03 Apr 2019 19:57:21 GMT

I will reply to your new question posted at Is it Possible to Collect Accurate Z Coordinate Values with Collector and Trimble R2. I will also be deleting my response below and moving it to your question.

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

AlinaTaus — Wed, 10 Apr 2019 13:43:53 GMT

Hi Lance,

First, I wanted to thank you for taking the time and sharing your knowledge on this topic. I too have been struggling for a few months now to get our set up right, so it's somewhat comforting to know I'm not alone

We bought the EOS Arrow Gold units his last fall and had the setup you've described in the other post where I had to run postprocessing to get the Z elevation. Within the last few months EOS came out with support for GEOID12B and Collector Aurora now supports z-enabled features so I had to redesign my workflow. We also updated our orthoimagery and now the basemap we are using is in a different coordinate system than the rest of my GIS data. I am wondering if you can offer some advice on my setup. Here is what I have:

The basemap is a hosted service by the State of Maine and it's in GCS NAD 1983 2011. It is also projected to NAD 1983 2011 UTM Zone 19N (meters)
We will be using the State's RTK, which outputs data in GCS NAD 1983 2011.
Our existing GIS data is in NAD 1983 StatePlane Maine West FIPS 1802 (feet)

The first question I have is, when setting up my z-enabled feature class for data collection, I have to set both a horizontal and the vertical coordinate system. I know my horizontal has to match my basemap, but it is the geographic coordinate system or the projected one? Second, for the vertical coordinate system I have to select what the EOS receiver will output, so this will be NAVD 1988 (meters). Now, for the Location Profile in Collector, I would think that I wouldn't need to set one up, since both the GNSS receiver and the basemap use the GCS NAD 1983 2011. If this is correct, then the only post-processing I would have to do is to reproject the feature class to the State Plane coordinate system and change the vertical to NAVD 1988 (feet) to convert my elevation from meters to feet. Does this sound right? Am I missing a step?

Thank you very much!

Alina

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

LanceCole — Thu, 11 Apr 2019 02:02:56 GMT

Alina,

I am not going to be able to help much. I have not had the opportunity to try the new z-enabled capability of Collector for ArcGIS (v19.0.1). This is primarily supported with the Arrow/EOS receivers and software and we are using Leica receivers that, unfortunately, have not caught up yet. Your workflow outline appears to be sound based upon my understanding.

The main issue with data obtained from Collector is "What coordinate system is my data in?" Going from a GPS receiver, running on various RTK systems, on different coordinate systems, that each manufacture integrates differently with Collector, pushing to AGOL, that uses base maps in one coordinate system but allows you to publish your features in another, back to ArcGIS Pro that does not clearly define which coordinate system it is currently working in, and the fact that there are hundreds of possible combinations of coordinate transformations, all to have your data not appear where you think it should be appearing, or is the data right and the map/image wrong. No wonder it is confusing.

One thing I can highly recommend is to create a set of control points. I have two sets I use, one is a small number of points around the office neighborhood that I can quickly go for a walk, on a nice spring day, and test a new collector map. The second is about 24 points over a large area that I can use to test at finer level that can catch things such as datum shifts that may not be visible on a small localized set of points. We collected these control points using a survey grade GPS, using the coordinate system we ultimately will be working with and typically using 5-10 minute averaging at each point. The points have been collected at several different times and the results averaged. I keep them in a shapefile or GDB feature class that is readily accessible.

When I am working with a new Collector Map or trying to figure out is my new transformation workflow model working correctly, I go for a walk and collect these points. Once collected I bring only the points into an ArcGIS session, perform my transformation workflow and then bring in my controls points to see how they compare. Sometimes my collected data shows up in the middle of the Indian Ocean, other times it is a few feet off but most of the time it is within the accuracy of the GPS. When it does not work I tweak my workflow and run it again, check my map or GPS configuration, recollect the points and check it again. It is simple and quick and I am checking against a single absolute known.

Working with multiple coordinate systems in you data is difficult. We are in the process of migrating from NAD 83(86) to NAD 83(2011) the difference between the two systems can be 2+ feet. The key here is to make sure you have the correct transformation assigned in ArcGIS between the two systems. There is also a performance price to pay for working with multiple coordinate systems as some of your data needs to be projected on the fly. This is most notable when it is your imagery that is being corrected. The best option is to work in a single system or as a second option work in your imagery coordinate system to minimize processing time. Another option, if possible is to obtain a physical copy of the imagery, is to reproject the imagery to your preferred datum. Just plan on some serious processing time depending upon the extent and image quality.

Sorry I cannot specifically answer your question. I hope this helps.

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

AlinaTaus — Thu, 11 Apr 2019 13:16:11 GMT

Lance,

Thank you for responding. I had plans to go out this week and find some of the survey benchmarks located around town and try to collect their location using my setup but we got 4" of fresh snow dumped on Monday, so will have to wait a bit... Ahhh winter! Anyway, another issue I discovered yesterday was that the web layer you publish to ArcGIS Online will be in the coordinate system of your data frame, so you have to make sure that the spatial reference of your data frame matches the one of the feature class you are trying to publish. I noticed this when I tried to bring the web layer back in Pro to look at the data I had just collected and saw that the spatial reference was in my local State Plane and not UTM Zone 19N. However, the data I collected was remarkably close (both XY and Z) to the data I had collected previously with a Trimble unit. I can't wrap my head around why I was able to get good data (I think!) with a wrong setup, but it just makes me even more nervous and less confident that I can do this on my own.

I do have a copy of the recent imagery and my original intention was to reproject it and cache it on our server (like we've done with some older imagery) but I ran into all kinds of error messages and had both the clip and reproject geoprocessing tools spinning for hours with no progress so finally I just had to give up and move on. I might go back to it at some point...

Anyway, I'm going to try and fix the coordinate system, go out and collect some benchmarks and hope that I'll get some accurate data! Thanks again for your input!

Alina

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

MitchJohnson — Mon, 06 May 2019 21:12:21 GMT

Hi Lance,

I believe what you have described here is the exact issue i have been facing-- the primary problem being understanding datum shifts and ESRI's projecting on the fly. I have two questions that maybe you can shed some light on:

1. Testing against NGS Survey Control Monuments. This is where I first noticed accuracy issues. When comparing your collected data, would you compare against the GIS files for these NGS Survey control points as they are, or would you re-project them to the coordinate system that your data is in (not your collector basemap projection, but the coordinate system that your feature classes are in). I want to trust the Survey monument GIS, but does it need any re-projecting if it is in a different projection than my data?

I am using a real-time differential correction service and am believe i know which coordinate system the map and the GNSS should be in, respectively, when setting up a profile. But what about projection changes comparing the data to control?

2. You mentioned that there are hundreds of potential datum transformations. Is the only way to discover the correct transformation to test each of them in the field?

Thanks,

Mitch

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

LanceCole — Tue, 07 May 2019 02:12:56 GMT

Oh the joys of data transformations.

1) you needed to lookup the data sheets for NGS Survey Control Monument you are using. These will list the coordinate system and datum used for the published data. They may be different depending when the monument was set and/or last verified.

2) make sure you are collecting the correct data using your equipment. I am fortunate in that I have access to in-house survey control data specific for my location in a known coordinate system and datum. Check with local surveyors, county map office for survey pin data on plats, state monument boxes for roadways, etc. You only need a few points to cross check against.

3) selecting a transformation - I keep our collected data in the datum we work with for our GNSS, in our case NAD83 (2011). I transform the control data to this system for verification. This can be difficult but typically common sense prevails. You know what system your data is utilizing and what system your control was collected with, what is the most logical path between those two. I prefer ArcGIS Desktop for the aspect of determining as you can load your data in and set the data frame coordinate system along with transformations for other systems. The transformations can readily be changed in Desktop whereas I have not found this functionality in Pro.

Once you have your data loaded in a MXD add your control data and play with different real-time transformation to see what works best. In our case our control data is in NAD83(1987) but our collector works with NAD83(2011). We found a combo transformation of NAD83(1987) to NAD83(HARN) to NAD83(2011) worked the best.

4) signs of a transformation issue

a) all your data is shifted the same distance and direction, within the accuracy of the data

b) a gradational shift such as data to the north is shifted NE, central data appears to be accurate and southern data appears to sifted to the SW. Another example is data to the north appears accurate but as you look at data further to the south a larger shift is noted in some direction and increases the further south you go.

c) some pattern in the data differences

5) accuracy issues - if the differences between your data and the control points are random most likely you are not looking at a data shift but rather an accuracy issue. You need to know the limits of your equipment and the control data. Keep in mind if your equipment is rated accurate to 6 or 12 inches that is typically a plus or minus. You need to double those values and not be looking for cm accuracy.

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

MitchJohnson — Tue, 07 May 2019 21:13:54 GMT

Thanks, Lance.

On your points:

1) Is it better to trust the lat\long from the datasheet and pick my own projection for bring in to GIS or use the shapefile download from the NGS.NOAA website? There's about a 1.5 foot difference between specifying a North American 1983 HARN datum myself, versus using pre-existing shapefile.

2) I assume you mean use some proper control points? I have found a few NGS monuments in the area that our surveyor has said are reliable for testing.

3) Because I am using Collector, the location profile requires a selected transformation between the map coordinate system and the GNSS coordinate system before any data can be collected at all. Identifying the transformation is the core of this problem. If i select additional transformations in ArcMap to the already collected data (for which a transformation was specified in Collector), wouldn't I be further abstracting the accuracy by "double-transforming"?

4) I am absolutely facing a transformation issue since point a) that you made is exactly what I've been seeing.

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

LanceCole — Tue, 07 May 2019 22:56:46 GMT

1) can you please provide the link to the datasheet you are referencing. I would like to use something you are actually utilizing to expand my reply.

2) if your surveyor has a survey grade GPS rover pole, have them shoot a series of points over an area that is readily accessible. I like the center of manhole covers because they are easy to find, big, do not move and can often be seen in imagery. Shoot how ever many locations you would like to use as check points using averaging for a few minutes at each location. Repeat on another day and average the results. You now have a known set of points, in a known coordinate system that you can check your data transformations against.

3) the GNSS metadata that is stored with your collector data contains the latitude, longitude and altitude (height above the ellipsoid) in the coordinate system your GPS is running. I use this data and project to coordinate system were are using. I am really looking forward to Collector support for 3D points using our GPS.

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

MitchJohnson — Fri, 10 May 2019 19:18:48 GMT

Emil GPS

PID: DG4231

Datasheet URL: https://www.ngs.noaa.gov/cgi-bin/ds_mark.prl?PidBox=DG4231

Gene 2

PID : OM1237

Datasheet URL: https://www.ngs.noaa.gov/cgi-bin/ds_mark.prl?PidBox=OM1237

Edina Taylor GPS

PID: DG4236

Datasheet URL: https://www.ngs.noaa.gov/cgi-bin/ds_mark.prl?PidBox=DG4236

2) I can shoot manhole covers, but how do I control for potential errors/shifts in the imagery?

3)OK, that is helpful. So, in my case it is storing the data in the GNSS coordinate system.

Thanks very much for helping with this, Lance!

-Mitch

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

AllisynHudson_Dunn — Wed, 19 Jun 2019 01:24:27 GMT

Hi Lance.

I am not sure if you are still actively monitoring this thread, but I am impressed with your knowledge and thoroughness in your testing and collection workflows and thought I would post my question in the case that you are.

I am currently transitioning our inspectors from Trimble TerraFlex to Esri Collector. To ensure we limit the amount of potential error introduced due to unnecessary transformations, I want to make sure I am setting up the profile and the project correctly. Here is the situation that we are facing:

Client DB is in WGS84
Client wants our deliverable in NAD83 because the RTK network we are using collects in NAD83
Once the client receives the data he then performs another transformation to convert the data back to WGS84
The client uses Esri basemaps, as do we.
Our current profile is set up as follows
- GNSS Coordinate System -- GCS NAD 1983 2011 (6318)
- Map Coordinate System -- WGS 1984 Web Mercator Auxillary Sphere (3857)
- Datum Transformation -- WGS_1984_(ITRF00)_To_NAD_1983_2011
We are using Trimble R2 units and Iphones
We are looking for subfoot accuracy

I have been testing out different scenarios but am not seeing the benefit of any of them. My main concern is how all the transformations will effect the final collection product. Considering the above bulleted info, should my collection database be set to NAD83, or mimic the clients setup in WGS84? It is my understanding the Trimble R2s collect in WGS84 by default, how does this affect all the other moving parts including the fact that the RTK is in NAD83? Also, what are your thoughts about our client's existing workflow? Should the data just be delivered in WGS84 rather than undergoing another transformation?

I apologize for any confusion, my head spins just trying to wrap my head around all the transformations that are occurring in this process.

Thank you for any advice you can provide.

Allisyn

Re: Cordinate systems and transformations with collector and high accuracy GNSS GPS - best practice

LanceCole — Wed, 19 Jun 2019 13:40:22 GMT

Hi Allisyn‌,

Two quick questions and I will work on a reply later today.

1) When you say the client wants the deliverable in NAD83 is that NAD83(86) or NAD83(2011). Also is this GCS or on a different coordinate system such as a State Plane system

2) Are you interested in XY data only or Z data as well?