Calculations on Geopoints

22344
58
06-08-2016 03:32 PM
IsmaelChivite
Esri Frequent Contributor
7 58 22.3K

[Updated February 22, 2019]

NOTE: This content is now obsolete. Please refer to this blog post for more up to date content.

The calculation column in XLSForms allows you write expressions that, when evaluated, set the response for a particular question. Calculations are supported in most of the questions types such as integers, text, date etc  For example, the Calculation expression below will sum the number of girls and boys in a class, so the field user does not need to enter the total number.

 

CalculationSurvey123.png

Calculations are pretty straight-forward once you understand the operators you can use. This help topic has a nice table with all the operators supported. I also really like Hanna's Blog Post

 

It may not be obvious how to use Calculations against geopoint questions, but it is possible!  Calculating a geopoint will literally move the map to the computed location. For example, say you have a list of wells and you want the user to select one of them, to set the X/Y coordinates of the well in a geopoint question.Geo_Smaller.gif

 

To Calculate a geopoint, the most important aspect is understanding how geopoints are internally represented in Survey123 for ArcGIS. The format is simple: a space-separated list of valid latitude (decimal degrees) and longitude (decimal degrees). For example, this is how a geopoint question, under the covers, is managed within the Survey123 app:

 

 43.221 -117.342

 

A geopoint object requires at least the latitude and longitude, but additional values can be added for its elevation (in meters), horizontal accuracy (in meters) etc

 

 43.221 -117.342 34 2

 

Of course, that is not what the field user sees, because they see a map... but you need to know how a geopoint object is stored if you want to work with Calculations and Geopoints. If your calculation generates a string like that, the map in your geopoint question will center itself there. You can also set the value of a geopoint question using the default column.

 

Ok, so lets look at this step by step through an example. Pretend we want to build a survey where the user selects a city from a list and have  the map automatically center at that city.

 

The location of every city is encoded in the cities choice list. Note that the coordinates of their location are in the name column. I separated coordinates with an underscore because spaces are not allowed in the name column.

 

 

The calculation expression parses this value to extract and format the data to be passed to the geopoint question. For that, the substr function is used. I had to put care into ensuring that the different values encoded started and ended at fixed positions in the string. That is, I reserved the first 10 characters for the latitude, then I added the separator, and then I left the rest of characters for the longitude etc If needed, I would add extra zeros at the end to make sure the coordinates had exactly the expected number of characters. There are some functions like split, that would theoretically help us parse strings with more flexibility, but at the moment they are not yet supported in Survey123. To properly build the string representing the geopoint object, I used the concat() function.

 

type name label calculation choice_filter

select_one continentcontinentContinent  
select_one citiescitiesCity continent=${continent}
geopointlocationMapconcat(substr(${cities}, 0,10)," ",substr(${cities}, -10)) 

 

Encoding the coordinates in a choice list will  work well for small numbers of points. Storing the coordinates in a CSV file and using pulldata() to get them is recommended if you plan to work with large lists.

 

Survey123 Connect for ArcGIS includes two sample XLSForms that will help you play with the  concept of calculating geopoints. To open the Samples follow these steps:

  1. Open Survey123 Connect or ArcGIS
  2. Click on New Survey
  3. Select the Samples category from the dialog and look for the Calculate Location from Choice List, or Calculate Location from CSV

 

 

 If you would like to block field users from overwriting the location set by your calculation, you can set your geopoint question to read-only.

       

So far, we have learned how a calculation can set the value of a geopoint, but you can also do the reverse. That is, you can use a calculation to extract data from a geopoint question, such as the X,Y and Z values and use them in your own expressions.  This technique uses the pulldata() function and it is described in more detail in thehttps://community.esri.com/groups/survey123/blog/2016/11/03/extracting-information-from-geopoint-que... blog post.

58 Comments
GlenGardner1
New Contributor III

Using the example of the wells in the choice list. How would adding new wells be handled? Would the survey have to be republished and redownloaded each time the choice list is updated? Can the choice lists be set to a related table instead of a geodatabase domain? If that is possible then a user can setup a survey to collect new features and then have additional surveys for follow up visits to the same features. This would be very helpful with how my organization conducts its field work.

IsmaelChivite
Esri Frequent Contributor

Hi Glen. Thanks for your comment.

We are designing some features in Survey123 to support workflows like the one you describe.You will probably be familiar already with the Sent, Outbox and Draft folders in the Survey12 field app. They resemble the way you work with e-mails, where you can save an incomplete Survey as a Draft, to send it later. The Sent folder shows you all Surveys you have sent, and the Outbox displays Surveys ready to be submitted... So the idea would be to add an Inbox folder, which would sync with your Feature Service. The Inbox would be populated with Surveys which you can open to edit and send back to the Server.  In your example, you would be able to configure your Survey to fill the Inbox bringing all or some Surveys based on a query. Then you will be able to search for Surveys in your Inbox based on a unique identifier (Instance Name) or even a map.

The other idea is to allow lists, like the list of wells in the example above, to be dynamically populated from a file or Feature Service query. Think of this as pulling data into your Choice List. This is something which we are also considering for the future.

SARADOMÍNGUEZ
New Contributor

Hi! Will be possible doing geotracks and geoshapes in Survey123?

Thanks

zkovacs
Occasional Contributor III

Not sure if I'm doing something wrong, but I cannot access the sample XLSFile mentioned above. Every time I try to access it I'm getting the below error. (Regardless I'm logged on to geonet and/or survey123.arcgis.com.)

Zoltan

IsmaelChivite
Esri Frequent Contributor

Thanks for reporting the issue downloading the XLSForm.

I changed the URL in the text above. It should be working now for you. Thanks!

zkovacs
Occasional Contributor III

Thanks Ismael. You may want to update the link at the bottom of your article too as it still points to the old location (the first link works ok).

Zoltan

zkovacs
Occasional Contributor III

Hi Ismael

I started looking into using calculations on geopoints as I was designing a survey. I used the same approach as detailed above, but I noticed something weird so I double checked it using the XLSForm provided above (which is also included as a template in survey123) and it does the same. I noticed that the geolocation question does not honour the manually set location as your last line suggests above:

"One last tip: If you would like to block field users from overwriting the location set by your calculation, you can set your Geopoint question to read-only."

Although it lets me modify the location, it pretty much ignores what I set manually and defaults back to the location set by the calculation. The co-ordinates displayed in the geolocation window are the same as set by calculation... I tested it on Windows 7 and Android and observed the same issue (can check in Survey123 connect too). Looking at the data that goes out when synchronising/submitting, it honours the co-ordinates set by the calculation and ignores the location I set manually. It gives you the impression that you can modify the location, but it does not overwrite the values/co-ordinates set by the calculation.

See screenshot of the process below.

Zoltan

Geolocation-Calculation_issue.png

IsmaelChivite
Esri Frequent Contributor

Thanks a lot Zoltan. This is not behaving as it should. I will log a bug on this one.

zkovacs
Occasional Contributor III

No problem. Hopefully it can get fixed in the next release.

IsmaelChivite
Esri Frequent Contributor

A fix was  implemented so user-defined locations would not be overwritten by the calculation. This fix will be made available in our next update to Survey123, which is scheduled in September 2016.

zkovacs
Occasional Contributor III

That's great news, looking forward to it. Thank you Ismael.

EduardoVasquez2
New Contributor II

Greetings. I need to do just the opposite, extract the "X" and "Y" coordinates from the geopoint field into two string fields called Lat and Long. I haven´t problem calculating the latitude using the substr function:

substr(${ubicacion},5,15)

But I can´t extract the Longitude because the number of decimals captured by the geopoint for X and Y are variables so I don´t know what numbers put into the substr function.

Does already exists another way to do this?

Thank you .

Eduardo Vasquez

ICA-Uruguay

zkovacs
Occasional Contributor III

Eduardo

I couldn't find any XLSForm operators to do this reliably, I think this needs to be done during post-processing. Lat/Long coordinates can vary in length depending on the location:

(There won't be a general setup with functions/operators since the values are variables as you mentioned and their length vary. There might be setups that work locally on a project though.) You could split the location string and use left/right/mid/find functions to extract the data, but a simple calculate geometry in ArcMap using the Shape field would be easier I guess...

I hope this helps.

Zoltan

IsmaelChivite
Esri Frequent Contributor

The issue around geopoint calculation overwrites is now resolved in Survey123 version 1.7.

More details about this release: International Day Against Nuclear Tests Release (1.7) 

zkovacs
Occasional Contributor III

Thanks Ismael. I'll have a look at it later on.

Update: I tested it and it now honours the value when I overwrite the geopoint calculation. On the other hand, have you changed the way to get the location co-ordinates? Before the update a simple calculation - ${location} - in a note field for example resolved the co-ordinates. Now it does not work at all, nothing is displayed. (Even if it's just a simple geopoint question without any calculation on it.) I tried it in a new survey created from a template in Survey Connect 1.7 , and the issue is the same. I had a look at the documentation, but couldn't find any updates on this part. Any ideas?

XLSForm:

EduardoVasquez2
New Contributor II

I have the same problem described by Zoltan. Previously we extract the coordinates using the "substr" function, but now we only get the expression: [object Object] when we try to extract the coordinates. How can we extract or obtain the coordinates?

Thanks.

IsmaelChivite
Esri Frequent Contributor

OK. We will need to look into this. We have to different issues:

  1. I think that using substr to extract coordinates from a geopoint is not reliable, because the length of the X,Y coordinates is not constant.  Somehow, we need to be able to get the X and Y coordinates of a geopoint better. At this moment we do not have a good solution to this. Lately we have been discussing in the Survey123 team the idea of using the pulldata function as follows: pulldata(“@geopoint”, ${location}, “z”). We are still thinking if this wold be a good approach, in which case, we will make this available to you.
  2. The second issue relates to the fact that the note is not showing any value anymore. It was unexpected to me that this behavior changed. Let me do some more research on this. I or someone else from the team will get back to this topic.‌
Nicole_Ueberschär
Esri Regular Contributor

I just experienced the same. I was expecting to get at least "something" when putting the result from the geopoint field into a note but not "nothing".

My client is very disappointed that his approach is not working any more. He also used the substr method. 

Of course it would be much easier if we could just use x, y, z instead of substrings... 

Would be great if you could provide a solution soon. 

AndrewHargreaves
Occasional Contributor II

Hi Ismael,

Using the "Calculate Location" sample in Survey123 Connect is there a way to autopopulate the continent and city based on the XY gathered from the GeoPoint? eg: my current location auto populates "North America" and  "Pennsylvania".

Thanks

IsmaelChivite
Esri Frequent Contributor

Nicole Ueberschär‌  Eduardo VasquezZoltan Kovacs

Here is an update on the issue you describe with geopoints in 1.7:

Using ${mygeopointquestion} to extract a geopoint value as a string should never have worked in previous versions of Survey123.  I would not have done differently than you: if something works for me I typically go and use it.  So what do we do next?

Technically for us it is very problematic for us to help you go back to a previous version (technically not possible in iOS and a real rabbit hole for us on Android).  This would not be a long term solution either because as soon as you update to 1.8, 1.9 or whatever is coming next will break your surveys again.

We are working on a quick fix for you to get things into a reasonable state:

  • The correct way to get a geopoint as a string will be string(${mygeopointquestion}).  This will start working in our next update.
  • In practice, to get this going you would need to:
    • Update your surveys to explicitly cast geopoint values into a string (using string() as described)
    • Publish the survey again (will not cause you to lose  data)
    • Ask end-users to update to our next update AND download the survey again

I totally understand this may not be your desired flow: If something worked for me in the past I expect it to keep working... but unfortunately we run into a dead-end situation.

Looking ahead, we also need to discuss what exactly you are trying to achieve by extracting the geopoint  values as strings.  If your end-goal is to parse the string to get the X,Y or Z values using substr, then we are in trouble too because while it may have worked for you, and unless you are  using some unknown magic, your XLSForm expression is bound to give you trouble soon:

  • The number of digits in a latitude/longitude pair is not guaranteed to be fixed. Sometimes you get more  digits, some fewer.  For this reason,  you cannot reliably extract the X,Y values.

As part of the quick fix, we are exploring the possibility of extending the pulldata function to support extracting the coordinates from a geopoint in a safe manner. It will look something like this:

  • pulldata(“@geopoint”, ${location}, “x”)
  • pulldata(“@geopoint”, ${location}, “y”)
  • pulldata(“@geopoint”, ${location}, “z”)
  • pulldata(“@geopoint”, ${location}, “HorizontalAccuracy”)

We are working hard to try to get all the above sorted out soon for you. In order to expedite the fix, we will email you so we can work together on a resolution to this problem.

Thanks a lot for so quickly report the issues you found.

zkovacs
Occasional Contributor III

Thank you Ismael, I really appreciate the efforts to get this fixed. Although it is not a deal breaker for us at the moment, I'm sure the capability will crop up in a project sooner than later. Happy to assist as much as I can, please feel free to get in touch.

deleted-user-23WYWUIp9HyE
New Contributor II

I'm testing the pulldata geopoint functionality using the XYZ entries from the Calculate Location From Geopoint sample form in Survey123 Connect as a model for rows in my XLS form.  I'm using an iPhone 6S and none of the fields (X, Y, or Z) populate in my form once I select my location in the geopoint.  Is there something I might be I'm missing here?  I called my geopoint 'Geopoint' in my form as opposed to 'location' in the sample form and I've adjusted the text in the calculation field to reflect this.

Thanks,

Tom

Nicole_Ueberschär
Esri Regular Contributor

Hi Tom, could you share your excel file for us to test it?

deleted-user-23WYWUIp9HyE
New Contributor II

Hi Nicole,

my excel file is here:

Dropbox - 13356_RochesterNH_MS4_v4.xlsx 

Thank you for assisting me.

Tom

AndrewHargreaves
Occasional Contributor II

@Thomas Hamill

It looks to me that there aren't enough fields in your pulldata*() calculation - in my experience it requires:

pulldata('CSVFileName','MatchingFieldNameinCSV','ValueFromCSVIwanttoPullBack',${FiledInMyForm})

deleted-user-23WYWUIp9HyE
New Contributor II

Hi Andrew,

I was using the XLS rows from the Calculate Location From Geopoint sample form in Survey123 Connect.  Do you know what the appropriate CSV file name would be to set up these fields so that they populate X, Y, and Z values from the geopoint or device location?

Thanks,

Tom

zkovacs
Occasional Contributor III

Tom

One of your calculations seems to be incorrect:

pulldata("@geopoint",${Geopoint},"accuracy")

should be (Link)

pulldata("@geopoint",${Geopoint},"horizontalAccuracy") 

Apart from that I don't think anything is wrong with your location calculations. I just tested a survey I used successfully on Android last week and it does work for me on iOS. I quickly published yours too and it works just fine. Have you updated your survey123 to the latest version (1.8.21)? If not, please do so as pulling location data has just been introduced in the latest version.

I hope this helps.

JennyferSarmiento
New Contributor II
JoelMeier
Occasional Contributor II

I'm having an issue where this works great in survey123 connect on desktop but as soon as I publish and try the survey online I can no longer calculate on the geopoint. Is there something that needs to be done to ensure this is able to be used online?

MichaelKelly
Occasional Contributor III

Hi Joel,

I don't believe pulldata() is currently implemented as part of the web forms just yet. So this will only work through the Survey123 app.

Michael

JoelMeier
Occasional Contributor II

Thank you for the reply Michael, I am using the substring method used within this excel sheet to put the data into the geopoint. It works with no issues on both android and survey123 connect but does not work on web forms. Is that also not supported in web forms?

JamesTedrick
Esri Esteemed Contributor

Hi Joel,

The web form has slightly more strict requirements for calculating geopoint than the field app.  The preferred way to do this would be to load the latitude and longitude into calculation questions via pulldata (supported on the web forms published by Survey123 Connect) and then use the concat() function to construct the location string in the geopoint question (the web form will not use + for string concatenation).

AndrewHansford
Occasional Contributor II

Hi there,

I think i am attempting to do the opposite here. I want to use the X/Y (Lat/Long) to test the nearest location and then populate the location with that information. It would also most be trying to perform a nearest neighbourhood function but using the calculation function.

Any thoughts on this?

Thanks

Andrew

JTedrick-esristaffichivite-esristaff

JulieRoebotham
New Contributor III

This is working great.. i'm doing this for surveys on known locations around the coastline of canada...i was hoping to hide the geopoint from the end user (not just readonly) and avoid having to make caches for areas where devices are offline, and just relying on the point being calculated from the lat long in the external csv. i *think* i've read everything and tested what i could - am i correct in that this is still not possible? (hidden doesn't seem to work as there doesn't appear to be an bind::esri:fieldType of geometry, and the appearance setting of 'hide-input' doesn't seem to do anything.).. would be great since we can calculate the geopoint to then just make the whole thing unavailable to the end user. Ismael Chivite James Tedrick

JamesTedrick
Esri Esteemed Contributor

Hi Julie,

You are correct, Survey123 does not support hidden geopoint questions at this time.

JenAmes
New Contributor III

Is this function still not supported by web forms? I got all of this to work in the app, but I'd rather be using a web form due to it's ability to update immediately with any form changes. Trying it on the web form though, it's not moving the geopoint to the selected location. Would I still have to use these functions you're talking about here, and if so, could you give me an example of what that text would look like in Connect?

JamesTedrick
Esri Esteemed Contributor

Hi Jen,

Yes, the webform requires the use of concat() to add (concatenate) strings together.  An example of this is in the Survey123 Connect Gallery in the 'Calculate Location from a CSV':

J_P_Montagne
New Contributor II

Thank you for this solution, Ismael, it works very well for me, except for the following snag: once I've sync'd my data to the Survey123 website, how do I get the field to display the label (e.g. "London, England") rather than name (e.g. +051.5000_-000.11670)? I have two location with the same XY coordinates but different labels (A and B). Viewing the data in Survey123 online and arcgis.online, and downloading the CSV only shows the name so I can't tell which entry it actually is.

My work around is to change one of the coordinates, for example A is +051.5000_-000.11670 and B is now +051.5001_-000.11670, but this doesn't really solve the problem.

Thank you!

JP

IsmaelChivite
Esri Frequent Contributor

Hi. J.P. Montagne  You may want to try the following:

Note that:

  • I added a hidden question called city were I will persist the label of the selected city.  The value is set with a calculation that uses the jr:choice-name function. More about this function at https://community.esri.com/groups/survey123/blog/2018/12/02/survey123-tricks-of-the-trade-xlsform-fu...
  • I also set the fieldtype for the cities question to null, to prevent these values from getting into the feature layer. This is optional, but I think it makes sense because after all, storing the coordinate string is useless.

To handle the situation where you have two entries at exact the same location, I suggest you use the pulldata approach, which is illustrated in the Calculate Location from CSV sample.  That is, instead  of getting the lat/lon values from a choice list, you will be getting them from a CSV file.

AndrewThompsonTRC
New Contributor II

Did any of the list options referenced above get implemented?

JamesTedrick
Esri Esteemed Contributor

Hi Nate,

The ability to dynamically populate a list is still an improvement we would like to bring to Survey123, though at this time we do not have timeline for implementing it.

AlexP_
by
Occasional Contributor III

Hello !

Would this work as restrict area to select from users? Please kindly confirm. Thank you.

Alex

JamesTedrick
Esri Esteemed Contributor

Hi Alex,

I'm not quite sure what you are asking.  If you are asking whether it can be used to prevent users from submitting a point in an invalid area, then it can with some limitations.  A good resource from the community (one of our distributor colleagues) is Limiting Extent in which a Point can Exist

AlexP_
by
Occasional Contributor III

James,

Yes, that is correct. Does it work on web browser? I read someone's comment that web browser now is work with limiting extent. Please confirm.

JamesTedrick
Esri Esteemed Contributor

Hi Alex,

Yes, the technique described in that blog post can be used in a web form.

JoelMeier
Occasional Contributor II

When using calculations on geopoints I notice that the point no longer defaults to the users position. Is this expected behavior and is there a way to overwrite it? 

JamesTedrick
Esri Esteemed Contributor

Hi Joel,

This is expected - each table can only store one point.  One way to work around this would be to have the calculated Geopoint occur inside a repeat and not have a geopoint in the main form section.

JoelMeier
Occasional Contributor II

Hello James,

I am not looking to have two points in one question but rather allow the user to select between using their current location or using a pre-calculated one. I would like the default to be their current location as is normal with geopoints, however when I do it as shown below it no longer uses the default.

JamesTedrick
Esri Esteemed Contributor

Hi Joel, 

If a geopoint has a calculation, there isn't a way to default to the user's location.  That being said, the user can select their location using the 'My Location' button:

JoelMeier
Occasional Contributor II

Is there a way to make the calculation only relevant if a specific previous answer is selected then?