Use images & AI to fill out a survey

[Kylie]

AI is a great tool for generating a first draft. Take a picture with Survey123 and let AI provide a first draft of survey answers, from an image description to choosing from a list of answers. Of course, as with most AI-generated content, it’s not a reliable final answer – you’ll want to make sure to review and update the answers as appropriate.

See it

Watch the survey in action:

In this survey, I included the prompt used to answer the question as the hint for the question. That’s not required; I did so here so you could see my prompts as examples.

You can fill out the survey from the video above yourself if you have an ArcGIS Online account. Use this survey to take a picture and see the information that is populated in the survey: https://arcg.is/1TbmWW2

NOTE: Using image analysis in Survey123 isn’t available when a survey is filled out by an anonymous user.

While all the questions attempt to use the image to generate a value (save the one verifying if you reviewed the answers), they don’t all always populate. AI can be a bit finicky. This is part of why all answers are reviewed before submission.

Make one

Follow these steps to make a similar survey of your own:

1. Enable analysis tools, including image analysis, for your organization and in Survey123. This must be done by an administrator of your organization. The following settings are required:
   1. Turn on "Allow use of AI assistants by members of your organization' in ArcGIS Online organization settings > AI assistants
   2. Turn off "Block Esri apps and capabilities while they are in beta" in ArcGIS Online organization settings > Security > Apps
   3. Select "Enable analysis tools" in Survey123 Organization settings > Extensions > Analysis tools
2. Create your survey in the web browser, including the image you want analyzed to populate other answers in the survey.
3. For the questions you want to fill out based on the image, add the question and click "Edit" for the calculation. In the "Edit calculation" pane that opens:
   1. Set Source to Question
   2. Under Extract property from question set Select a question to your image question.
   3. Set Select a property to Analyze image.
   4. Under What would you like to extract from the image specify the information to pull from the image. See "Writing the image extraction prompt" below for some tips on writing a good prompt for the extraction.
      
4. Add the other questions for your survey.
   You can have multiple questions that are populated by analysis of the same image using different prompts.
5. Recommendation: Include a question verifying if the person has reviewed all AI-generated survey answers. Marking that question as required means the survey won’t submit until the person marks that they have reviewed the answers instead of accepting what AI provided as the final answer.
6. Test your survey!
   Prompts often require a few revisions to get even better results.

Writing the image extraction prompt

Here are tips for writing the prompt that will be used to pull information from the image:

1. Ask for the specific information you need; be precise and detailed.
   Don’t ask what the image shows. Instead, ask for the information you need from the image.
   
   For example, the example survey used the following prompt to describe the image:
   

   Describe the recyclables in the picture. Are any items not recyclable? Of what is recyclable, what materials are there? Is something unique about them?

2. Be specific about the answer format you want.
   If collecting a count of items shown in the image, have your prompt include that you want a number returned. If populating a single select, specify when each single select choice should be selected.
   
   For example, the example survey used the following prompt to determine if the recyclables are indoors or outside:
   

   Is the image taken indoors or outdoors? Set the answer to "Indoors" if indoors, otherwise, set it to "Outside"

3. Iterate on your prompt.
   Put your first prompt and take some test images to see how your survey is filled out. Iterate on the prompt until you are getting closer to the answers you expect.
   
   
4. Don’t expect the same answers each time.
   The same image won’t always result in the same answers with AI. This isn’t something you’ve done wrong but is part of what to expect when working with AI.

A warning

I often find that populating fields other than a text field based on an image work sporadically. Sometimes uploading the same picture to the survey response a second time gets a number field that was empty after the first time to populate. However, deleting the image clears the fields that were populated based on analysis of it, so other answers will also be recalculated.

Some use cases

Here are some places analyzing images can be helpful in surveys. This list is by no means complete but is presented as a starting point for generating ideas.

• Describing what a picture shows to have a written record of it
• Filling out water quality values by taking a picture of a test strip and the chart for reading it.
• Analyzing if materials shown are recyclable or not.
• Reading street signs or other signs.

To learn more, see this Survey123 blog post about the Survey123 analysis tools.