Values of calculated fields not stored and editable in repeats

[EricaNova]

I have a survey that asks users to input the number of animals they see during surveys. Sometimes, surveyors want to enter specific details about certain focal animals, such as whether the animal had an eartag. In some cases, they know that they saw a group of animals, but were only focused on finding a particular individual, or family, within that group. Therefore, in the main part of the survey, I ask users to input the entire group size, and the number of focal animals they want to enter detailed information about.

Let's assume that we have a group size of 5, with only 1 focal animal, "George". This means that I want two records in my repeat: 1 for the focal animal, and one summarizing the remaining 4 animals. I have set the number of records in my repeat dynamically by setting "repeat_count" in my form to ${number_records}. In this example, ${number_records} is two. 

If the user is filling out the first record, I want them to answer a number of questions, such as whether the animal has an ear tag. I want to automatically calculate the value of the last record for the remaining 4 animals, to say that ear_tag_observed is 'unknown'. I do not want it to be null. I also do not want to set a default value for ear_tag_observed, because I want to ensure high data quality by forcing users to enter all required fields. I don't want a user to have to enter 'unknown' for non-focal animal records - it should be automatically filled out to save precious time in the field!

My solution, which works perfectly to store the data, is to have users select 'yes', 'no', or 'unknown' in the field 'ear_tag_observed_c'. This field is not stored. A second field, which is stored, says "if this is the last record in the repeat, enter 'unknown' into the 'ear_tag_observed' field, otherwise, make 'ear_tag_observed' equal to the user-entered value."

The only kink is that when users go to edit the data in the repeats in the web app, the values of ear_tag_observed are blank for all focal animals! This is because the calculated field is not stored (and I don't want to store it). 

I've tried to get help with ChatGPT, but keep running into issues with not being able to use expressions in the default field, or it wanting me to create dependency cycles.

Is there a way for me to get around this issue? Again, my requirements are:

• users should not have to enter data for non-focal animals - it should be calculated
• data value for 'ear_tag_observed' should be 'unknown', NOT null, for non-focal animals
• I don't want to store the calculate fields in my database
• data must be retained for editing in the repeat table for focal animals in the S123 web app.

Here's an example of how that 'ear_tag_observed' field is currently calculated in my survey:

type	name	label	hint	appearance	required	readonly	required_message	calculation	relevant	bind::esri:fieldType
integer	number_animals	Number of animals	The number of animals this record is summarizing.			yes		if(position(..) < ${number_records}, 1, ${number_remaining_animals})
select_one list_yes_no_unknown	ear_tag_observed_c	Calculate		horizontal-compact	yes		This is a required question		position(..) < ${number_records}	null
select_one list_yes_no_unknown	ear_tag_observed	Ear tag observed?	Only this field is stored.	hidden				if(position(..) < ${number_records}, ${ear_tag_observed_c}, 'unknown')

[LaurenceTait]

Not surprising you get dependency cycles. You want S123 to do something and not do it, simultaneously. You are asking the app to check if a field meets a certain value, while simultaneously asking it to delete the field.

Either S123 forces the user to enter the data into a field, or the field is automatically populated. If the field is automatically populated, whether by a value in “default” or a value in “calculation”, then you can’t force the user to enter a value into the field. You can either populate by default, or demand user input, you can’t do both

The sets “Focal Animals” and “Animals with Observed Tags” are exactly the same, and the set “Non-Focal Animals” is equal to the difference between “All Animals” and “Non-Focal Animals”.  However you don’t want to store information on whether an animal has an observed tag, but you want to store information on whether an animal is a focal animal. You can’t do that. If you store information on whether an animal is a focal animal, then you are storing information on whether it has a tag. And if you want to avoid storing information in whether it has a tag, you also have to avoid storing information on whether it is a focal animal.

There are a dozen different ways to accomplish this, but they all require that you store information that identifies focal animals.

You can store information on whether an animal has a tag directly. You can store it indirectly by requiring users to enter data into tag number fields only if the animal has a tag, or only if it doesn’t. You can do it in a complex manner by by storing the number of each repeat and and referencing that to the number of  “Non-Focal Animals” and the number of "All Animals”.

But you can’t refuse to store information on whether an animal has a tag, and also automatically populate a field conditional upon whether the animal has a tag. Either S123 has the information stored to identify if the animal has a tag, or it doesn't have the information stored to identify if the animal has a tag. There is just no way that you can calculate information that is specific to focal animals without providing information to identify focal animals.

[EricaNova]

Hi Laurence, thanks for your comment! The example I gave was a subset of my real data. In the real dataset, "focal" animals are any animals that are (1) marked, OR (2) of known age OR (3) of known sex. So, a focal animal does not always have a tag.

My workflow actually does work in the form to not have a default, and use a calculation, it's just the ability to edit I need. 

[LaurenceTait]

You may be missing the point.

“Focal Animals” is defined by trait "A". It doesn't matter if "A" is "Animals with Observed Tags”  or if "A" is “Animals with Observed Tags OR Animals with a known age OR animals of known sex”. The point is that the sets “Focal Animals” and “Animals with trait A” are exactly the same, and the set “Non-Focal Animals” is equal to the difference between “All Animals” and “Non-Focal Animals”.

You want to S123 to carry out an instruction if an animal is a Focal Animal, but you refuse to store the only information that identifies Focal Animals. Of course S123 can't carry out the calculation because it can't identify whether the precondition is met.

Your setup "works" in the form, the first time through repeat because the values are held in working memory while you are in an individual repeat. Once you upload, the values are, of course, lost from the working memory. Because you have chosen not to save the values, the app can't retrieve. At this point the workflow stops working.

You simply can not  get a machine to do something and not do it at the same time. It can't delete a value and simultaneously save it for future use. You have four options:

Store a value that identifies a Focal Animal.

Wipe the value that identifies a Focal Animal, in which case you lose any values calculated from that identifier data if you try to edit.

Lock the values, which precludes editing.

Request the machine to do a if b and also delete b. This generates a dependency cycle.

What would be the maximum number of focal animals that would appear in a group? If it less than 10,  and you absolutely insist on not storing data that identifies individual focal animals , then you will need to abandon the use of repeats and mimic the same effect using pages. Pages can reference an individual animal number which, provided we are told how many focal animals and how many total animals,  can be used as a method of storing whether the animal is focal or not. It's along winded and complex way of doing what could be done with stored tick box value, but that seems to be what you require.

[EricaNova]

Hi Laurence,

At the beginning of the form, users are asked for the group size (say, 10) and number of focal animals (say, 5).  The number of focal animals may be 0 to the group size. Group size and number of focal animals is stored outside the repeat. Inside the repeat, I always create a record for each for each individual focal animal, and the last record is for all remaining animals. So Survey123 does 'know' if each record in the repeat is a focal animal, or not. If I had to store a single field saying "focal animal? yes/no" that wouldn't be so bad.

It's fine if this isn't possible - I'll use a default choice if need be. I was just checking here to see if an alternate approach was possible here before using defaults.

Appreciate your time.

Erica

[LaurenceTait]

I must be missing something.

I'm fairly sure that I understand clearly what you are trying to achieve, which is why the attached XLSX achieves it, albeit without using a repeat.

How are you storing information in Survey123 that indicates whether a record within a repeat applies to a Focal Animal?  Based on what you've written, the only information that could indicate whether an repeat record is a focal animal is whether it isn't the the final repeat record. But you aren't storing information anywhere that records the number any repeat record.

In your example above, when you are editing repeat 6, the value for "repeat number" will be 6, and all values dependent on "repeat number"  will be calculated based on the value 6, which is equivalent to a non-focal animal. When you edit repeat 5, the value for "repeat number" will be 5, and all values dependent on "repeat number" including those in repeat 6  will be re-calculated based on the value 5, which is equivalent to a focal animal.

In short, "repeat number" has a value equivalent to the repeat currently being edited/viewed. That is a deliberate and necessary choice. But because it varies when editing a repeat sequence, it doesn't store any information. Yet the repeat number seems to be the only information that in any way informs S123 whether a repeat record is a focal animal.

You could put a count into your repeat with a 'once()' function, and that would store information pertaining to whether a record is in the final repeat. But you aren't doing that, and doing so would be more complex, more time consuming and require more storage and processing power than simply remove the 'null' from you current {ear tag observed c} field.

You can intend to make a record for every individual focal animal, but you need to pass that information to S123 somehow if you want it to perform actions contingent on that information. And you don't seem to be doing that in any way that is invariant - you aren't storing the information that tells S123 that you will create a record for each for each individual focal animal + one additional record for all remaining animals. You know that is what you intend to do, but you aren't communicating it to the App in any way that is being stored.

Can you tell me  what stored field you think is informing S123 that a repeat record is for  a Focal Animal? Once we have that, it will be easy to do what you want.