Pulling data from geopoint questions

35271
26
11-02-2016 09:02 PM
IsmaelChivite
Esri Notable Contributor
14 26 35.3K

[Last Updated February 23, 2019]

In a previous post, we explored how expressions in the calculation column of your XLSForm could be used to programmatically set values on a geopoint question. In this occasion, we will do the opposite. That is, we will describe how you can extract data out of a geopoint. Here are some good reasons why you may want to do this:

  • Store geopoint properties as attributes: Often times, keeping the X/Y coordinates of a point encoded in a geometry field is not ideal.  By pulling data out of a geopoint question you can store the X/Y coordinates in specific fields. You can also extract and store other values from your geopoint properties such as the elevation, speed or horizontal accuracy if present. Having all these values as attributes can help during QA/QC as well as data exchange processes.  Storing metadata about your locations is particularly important if you are  using an external GNSS receiver with Survey123, because all that metadata is critical to understand the quality of your data.
  • Build data validation rules: You can use geopoint properties in your form to create data validation rules in your form. For example, you can create a rule to warn users if a particular horizontal accuracy has not been reached, or if the location has been captured while the field user was traveling at an excessive speed.

What a geopoint is made of?

We all know that geopoint questions store location information, but what is really inside a geopoint value? At a minimum, unless a geopoint question is empty, you will always find the latitude and longitude of a location. Survey123 never works with projected coordinates: geopoint values are always in geographic coordinates using WGS84.  Now, you could potentially find much more than just a latitude/longitude pair within a geopoint value. The most common properties include:

Property NameDescriptionUnits
xLongitude, positive in eastern hemisphere, negative in western hemisphere

Decimal degrees

y

Latitude, positive in northern hemisphere, negative in southern hemisphere

Decimal degrees

z

Altitude, meters above sea level

Meters

horizontalAccuracy

Horizontal accuracy of the x and y coordinates

Meters
verticalAccuracy

Vertical accuracy of the z coordinate

Meters
speedGround speed

Meters per second

verticalSpeedVertical speed

Meters per second

direction

Direction of travel measured clockwise from north

Decimal degrees

magneticVariation

Angle between magnetic and true north

Decimal degrees

positionSourceType

Returns the category of the position source.  Unknown (0), User (1), System Location (2), External Device (3), and Network Device (4).

fixType

Returns the type of position fix the coordinate has. Potential results are NoFix (0), GPS (1), DifferentialGPS (2), PrecisePositioningService (3), RTKFixed (4), RTKFloat (5), Estimated (6), Manual (7), Simulator (8), and SBAS (9).

If a geopoint question contains all the properties above or a subset, depends on the following:

  • If the location is set by the end user by manually tapping on the map, then only the x an y properties will be populated. It is possible to also let the user type the elevation, which ultimately would populate the z property. 
  • If the location is set by the device, the number of properties with data will depend on the device and also on how that location was obtained.  The accuracy of the values found in each property is as well completely dependent on the device providing the location information to Survey123.

If relying on consumer devices to retrieve the location properties above, you should take values with a grain of salt. Your average iPad or Android smartphone could often give you a reasonable x,y and horizontal accuracy but everything else could or could not be trustworthy. In fact, many consumer devices do not even populate most of the properties above. I do not want to say consumer devices will give you useless geopoint  properties, but you should proceed with caution understanding through trial and error what devices and in what conditions will give you accurate information. 

By connecting Survey123 to an external GNSS receiver, not only you can significantly improve the location accuracy of your data, but also get many more properties associated with your geopoint values.  Here is a list of the most important properties you can get. The complete list is available in our help.

Property NameDescription
positionSourceInfo.deviceNameThe name of your external GNSS device

positionSourceInfo.deviceType

Indicates how your GNSS is connected to your device.  Unknown (-1), Bluetooth (0), Serial Port (1), and Bluetooth LE (2).
positionSourceInfo.antennaHeightThe user defined GNSS antenna height.
positionSourceInfo.altitudeTypeIndicates the selected altitude type: altitude above mean sea level (0) and height above ellipsoid (1).
geoidSeparationReturns the difference between the WGS-84 earth ellipsoid and mean sea level as reported by the GNSS receiver. This is also sometimes referred to as orthometric height.
accuracyTypeRMS (0) and DOP (1)
positionAccuracyReturns the mean radial spherical error. Encompasses both horizontal and vertical error.
differentialAgeReturns the age of the differential signal and correction used by the GNSS receiver to differentially correct the position.
referenceStationIdReturns the differential reference station ID (DSID) of the station used by the GPS receiver.
satellitesVisibleReturns the number of satellites visible at the time of location capture.
satellitesInUseReturns the number satellites being used to return the position data.

Reading geopoint properties using the pulldata function

All the geopoint properties described above can be extracted using the pulldata function. For example:

pulldata("@geopoint", ${location}, "horizontalAccuracy")

The "@geopoint" parameter  to the pulldata function indicates that values will be extracted from a geopoint value.

The second parameter indicates the question in your XLSForm that holds the actual geopoint value. You will want to make sure that the question you reference is actually of type geopoint.

The third and last parameter defines the geopoint property you want to extract. You will want to make sure the property name is included in the table above and make sure that you specify it with the proper capitalization: HorizontalAccuracy is not the same as horizontalAccuracy.

Learning by Example:

Here is a simple XLSForm that will extract the X,Y and horizontal accuracy of your geopoint and store these values as attributes. Using this technique, you have complete freedom as to which fields in your feature service will be  used to store the geopoint properties.  You use the Name column in the XLSForm to define which field will keep the data.

TypeNameLabelCalculation
geopointlocationLocation
decimalLatitudeLatitudepulldata("@geopoint", ${location},"y")
decimalLongitudeLongitudepulldata("@geopoint", ${location},"x")
decimalHorAccMetersAccuracypulldata("@geopoint", ${location},"horizontalAccuracy")

Realistically, I would never store geopoint properties as illustrated in the example above. Mostly because I would not want end-users to get distracted with questions in the survey that get automatically populated with these properties. I would not want end-users to modify these calculated values either. 

A more  refined approach would be to set  the Type of question as hidden. In this way, the Latitude, Longitude and accuracy questions will not be shown in the form, but their information would still be sent to the feature service on submit. To ensure  the actual values are stored as numbers in the feature service, I would also set the esriFieldType column to esriFieldTypeDouble as shown here:

TypeNameLabelCalculationbind::esri:fieldType
geopointlocationLocation
hiddenLatitudeLatitudepulldata("@geopoint", ${location},"y")esriFieldTypeDouble
hiddenLongitudeLongitudepulldata("@geopoint", ${location},"x")esriFieldTypeDouble
hiddenHorAccMetersAccuracypulldata("@geopoint", ${location},"horizontalAccuracy")esriFieldTypeDouble

When working with horizontal accuracy, you should also learn about the meaning of the accuracyThreshold column.

You can use very similar expressions in your constraints and relevant statements. In the next example, we will pop a message in case that the geopoint value was captured while moving faster than 0.2 meters per second.

TypeNameLabelRelevant
geopointlocationLocation
calculatespeedSpeedpulldata("@geopoint", ${location},"speed")
noteDo not move while fixing location!0.2<number(${speed})

Like every time you use pulldata() it is best practice to use this function alone. This is why I broke down my last example into two separate questions: One calculate question to invoke pulldata() and a separate question where I compare the output of the pulldata function with a number.

In this blog post I simply wanted to introduce the use of the pulldata("@geopoint") function. With it, you can build expressions to persist location metadata as GIS attributes, but also to create powerful data validation rules in your smart forms.

26 Comments
RobertMarros
Occasional Contributor II

Is it possible to do the same thing but with Geo-tagged Photos? Can I use the same pull data functionality to extract the EXIF metadata from a Geo-Tagged Photo? 

Thanks 

HamidYunus
New Contributor III

While this enhancement is not immediately applicable to anyone I'm currently working with, I can see how the Z value, direction, and speed can be so useful. Thank you for highlighting this. 

janispaulsen
Occasional Contributor

This worked, but only added the lat long to two decimal places.  How do I make the results go up to 4 decimal places?

IsmaelChivite
Esri Notable Contributor

Hi janispaulsen‌. I believe that you may be using a client application that is formatting the X/Y values to 2 decimal places. By default, your X/Y coordinates will contain over 10 decimal places.  If you are using ArcGIS Pro or ArcMap, ensure that the formatting of your X/Y Fields is not limiting the display to 2 decimal numbers.  You can also  control formatting of attributes in web map popups.

janispaulsen
Occasional Contributor

Ah, I was able to modify in the pop up format, thanks!

JillHalchin
Occasional Contributor II

Hi, Ismael,

This will be very useful as we get into testing mobile and external GPS this summer. I hope that you can get IOS devices to read the accuracy of the receiver soon.

I've noticed that the dashboard report for a single record shows only 4 decimals for the location.  I will definitely add the pulldata functions to the survey, so that should, if I'm understanding correctly, give as many as 10 decimals. Is it possible in the spreadsheet to limit the number of decimals to 5 or 6? 10 would be misleading.

by Anonymous User
Not applicable

Have you tried the round calculation for these fields in order to limit the number of decimals returned?

https://community.esri.com/groups/survey123/blog/2016/09/06/new-mathematical-functions-make-your-for... 

MC2
by
New Contributor

This use of the `pulldata()` function does not seem to work properly in the web client versions of a survey.

Using the approach above, the survey loads & works fine on mobile devices and in Survey123 Connect preview ...

The Web client, in contrast, shows an error:

Error loading survey. Error message: External instance "@geopoint" is empty., FormLogicError: pulldata with incorrect number of parameters found: pulldata("@geopoint", location , "y") Please contact survey author.

This makes sense, because there is no "external instance @geopoint" ... 

The geopoint doesn't exist until after the survey loads & gets user permissions to create a geopoint to reference...

Are there different implementations of pulldata() to explain the "incorrect number of parameters"?

The function signature explained in the post above only takes 3 arguments:

pulldata(<@geopoint>, <${someGeopointToUse}>, "aDesiredGeopointProperty")

...But the pulldata() function as explained elsewhere requires 4:

https://community.esri.com/groups/survey123/blog/2016/10/27/the-pulldata-function-access-external-da...

pulldata(<csvfile>, <returnColumn>, <lookupColumn>, <lookupValue>)

Seems the Web App is assuming the latter, and/or the former is leaving 1 out, but's not clear which one.

Any help or thoughts?

This is one of the major obstacles to reconciling the mobile version of our survey with the Web App.

Thanks!

JamesTedrick
Esri Esteemed Contributor

Hi,

Yes, the pulldata('@geopint') syntax is not supported on the web.  That version of the function isn't in the official XLSForm or XForm spec- it's a custom function we implemented in the mobile application as there is literally no way to reliably extract this information given the spec.  The web form is based on a project that implements the specification, so bringing the function into the web form requires additional customization on our part.  We haven't prioritized this yet as we think it will be a bit difficult to implement.

MC2
by
New Contributor

This would limit the result to only showing 5 decimal places, but it doesn't require 5.

The original problem was that the coordinate value was being truncated to only 2 digits, when it needed 4 or more.

If you're manually editing the coordinates on a map, you can use a constraint to force a minimum length:

string-length(${latitudePulledFromGeoPoint}) > 5

But otherwise there doesn't seem to be a mechanism for dictating precision/scale of the lat/long values (is there?)

MC2
by
New Contributor

Ok, thanks for the quick reply ...

Is there any way to determine the platform in the XLSForm so I could conditionally remove/apply the relevant constraints that are referencing the geopoint?

Or another way to constrain the geopoint's extent?

(I'm using pulldata() to determine x,y and then constraint inequalities to bound the acceptable input).

... Is there a way to catch/check if a value is null/empty/undefined?

Not sure if that syntax exists here.

Thanks!

Chris_Anderson
Occasional Contributor II

I'm checking to see if any progress has been made to solve the pulldata('@geopint') issue on the Web form?


Thanks

JamesTedrick
Esri Esteemed Contributor

i Chris,

We are planning to address this in 2.7, our next release.  That is currently scheduled to be released in late March.

SuddhaGraves
New Contributor III

Is there a way to pull geometry from one feature to another?  We have a Survey123 Form that uses related tables where there is a parent and child feature.  We want to create the geometry from the parent table and then pass it to the child record.  This way the geometry will be exactly the same between both the parent and child record.  I know how to use the pull data function to grab the lat/long, but I want to pull the geometry.

JamesTedrick
Esri Esteemed Contributor

Hi Suddha,

You can not pull the geometry directly; instead, you would pull the lat & long and then calculate that into the geometry.  Please refer to the 'Calculate Location...' samples in Survey123 Connect as well as Geopoints—Survey123 for ArcGIS | ArcGIS and https://community.esri.com/groups/survey123/blog/2016/06/08/calculations-on-geopoints 

AdamInglis1
Occasional Contributor

I'm not able to get the vertical accuracy to appear in my Survey123 forms using an Android phone (OnePlus 5t) on either the internal or external GPS receiver (Leica GG04 plus).  The horizontal accuracy field populates, but the vertical accuracy field is blank.  I have even tried using the sample GeoPoint form unaltered with the same results.

I've also tried iPads running iOS 10.3 & 11 with no success.  Using the vertical accuracy on the iPad with not drop below 19 m when both the Leica Zeno Connect app and Collector claim that the vertical accuracy is 1.68 m.  I assume this is an issue similar to iOS operating system will not reporting horizontal accuracy better than 5 meters, even if in fact your GNSS receiver will be giving you more accurate data than 5 meters.

Do I need to use Collector to drop a point and send the vertical accuracy over to Survey123?  This is not something I've tried yet but I need a way to record the vertical accuracy.

AlexP_
by
Occasional Contributor III

Hello team!

Would this work as restrict area to select from users? Please kindly confirm. Thank you.

Alex

RyanBohan
Occasional Contributor III
by Anonymous User
Not applicable

Hello!

 

We have a Citizen Hurricane Damage Report survey that allows citizens to search/find their address on the map and it adds a point.  Super helpful since it does the heavy lifting for geocoding and ensures we get a complete address from the user. However, when reviewing the actual report data I don't see the actual address listed anywhere just the lat/long coordinates.  Is there a way to have a map question and retain the actual address entered/found?  The tax department needs both the mapped point and the actual address entered.

 

Here is the survey: https://survey123.arcgis.com/share/d3efd0fd3fab4635ab9120faab5aa4ee 

 

It was created using the web designer.

 

Thanks!

JamesTedrick
Esri Esteemed Contributor

Hi Angelia,

It is possible to retrieve a reverse geocode of the address entered; see Geopoints—ArcGIS Survey123 | Documentation for more information.  It isn't possible to store values entered in the geocode/geosearch box, and it may not be value you desire (for example, someone may simply type their ZIP Code and then move the map to their house).  In your situation, I would recommend having distinct fields to store the written address as well.

by Anonymous User
Not applicable

I was afraid that was the answer.  Thanks so much for responding!

FlorentBigirimana
New Contributor III

I am wrong or Fix Time is missing among possible calculations ? 

JamesTedrick
Esri Esteemed Contributor

Hi Florent,

By "Fix Time", you are referring to the timestamp that the current reading was calculated on?  Yes, that is not available as part of the GNSS metadata with Survey123.  This could be a potential enhancement- could you register this in https://ideas.arcgis.com ?

by Anonymous User
Not applicable

After further testing of the map question in Survey123, it appears that it is using the World Geocoder.  Is there any way for a public survey to use our organizations's geocoder instead of the World Geocoder?  Or at the very least in addition to?  The World Geocoder doesn't find half the addresses in our county.  It is woefully out of date (by many years). Our geocoder finds every address in our county and has suggestions.  This is a deal breaker for using the map question for us.  Is there a way to use our geocoder and not the ESRI World geocoder?

NealeHutcheon
New Contributor II

Hi. We want to capture latitude, longitude in a public web survey opened from a QR code. It seems the (unskilled) operator needs to open the geopoint map and click on the 'target' icon to get this to work. Is there a way to hide and automate this more, even if the operator needs to accept a "location accept" permission request?

Thanks!

KrisRobbins2
New Contributor II

Is there a bug/issue that is causing the geopoint pulldata calculation to fail with this error message?  "There has been a problem trying to replace ${location} with the XPath to the survey element named"

I'm getting this error when I try using the geopoint pulldata calculation for Lat and Long auto population in my survey as found above.

Everything is set up exactly the same in my survey as it is in the example but I get this error every time that I try updating the survey from the XLSForm.  Any ideas on why this may be happening?  Thanks!