topic Re: Combining two feature layers for Survey 123 in ArcGIS Survey123 Questions

Combining two feature layers for Survey 123

SMW_GIS — Wed, 18 Oct 2017 19:08:10 GMT

Hi,

I was wondering if anyone has had any experience in combining two feature layers for Survey 123. For our client, we did condition assessments on their assets using Survey 123 with the form hosted on our organizational account. They have now taken the form and are hosting it on their organizational account for all future condition assessments (a few fields have been added to their form as well as choices in the choice lists). They want to combine the data we collected on our feature layer to theirs, so that all the data is stored in one place. Does anyone know if this is possible?

Re: Combining two feature layers for Survey 123

TL2 — Wed, 18 Oct 2017 19:25:06 GMT

Could you export your data to a file geodatabase, then use ArcGIS Pro to append to the destination table. That's what I would try.

Re: Combining two feature layers for Survey 123

ShanaGail1 — Tue, 01 May 2018 00:42:49 GMT

I'm trying the Append in AGOL but haven't tried ArcGIS Pro yet. So far it's hanging up on "Analyzing data..." and never completes. Did you try TL suggestion with success?

Re: Combining two feature layers for Survey 123

ShanaGail1 — Fri, 04 May 2018 00:56:18 GMT

Hi,

I discovered (with the help of an Esri Analyst) that the Append function to a hosted feature layer in AGOL cannot handle related tables. My survey had repeats, so I was out of luck going that route. We tried using the Append geoprocessing tool in ArcGIS Pro, and was able to update the parent table successfully. However, the Global ID was not preserved and the related tables were not updated.

The Esri Analyst logged these enhancement requests:
ENH-000113718 : Allow Workflow or ability for Survey123 to load, or append all records from a previous survey that included Repeats, i.e relationships in the Hosted Feature Layer.
ENH-000113717 : Add ability or workflow that would allow Append operations to preserve Global ID for hosted feature layers with relationships .

The Analyst suggested these workarounds:

-For Survey data without repeats/relates: Append In Pro to Hosted Feature Layer

-For Survey data with repeats/relates: Append in Pro using Preserve Global ID to Enterprise Geodatabase.

I hope this helps somebody else .

Shana

Re: Combining two feature layers for Survey 123

ShanaGail1 — Mon, 29 Oct 2018 20:24:13 GMT

Kristy Nicoll‌ Hi Kristy, I meant to update this thread... I did on a different conversation. At the UC I asked about this issue in a workshop and the person leading it (Paul Barker?) was able to do the append using a related table. So perhaps my issue had a root cause of something different. Just didn't want to send you on a wild goose chase!

Re: Combining two feature layers for Survey 123

Anonymous User — Tue, 30 Oct 2018 12:25:20 GMT

Thanks for the update Shana! I've been unable to append with related tables at all. Seemed like your solution would work for me but i've been getting a 999999 error when I use preserve global IDs. Going to open a case this morning in case there is something silly i'm missing.

Re: Combining two feature layers for Survey 123

ShanaGail1 — Tue, 06 Nov 2018 19:39:43 GMT

Thanks for pursuing it further. I wouldn't be terribly surprised if we encountered it again, so if you post the outcome here, that would be awesome!

Re: Combining two feature layers for Survey 123

GretchenFolse — Mon, 19 Nov 2018 22:28:50 GMT

We are running into a similar problem when trying to reload previously collected survey data to a republished Survey123 hosted Feature Layer. We're using Survey123 as our collector app for HazMat incidents (because the built-in Reports in Survey123 are pretty snazzy), and we have repeats built in for responding agencies, vehicles and chemicals involved, and for including photos of the incidents and the clean-up ops.

Using Append through ArcGIS Pro seems to work great for reloading all of our data to the republished survey EXCEPT the photo attachments. It creates the same number of placeholders for the photos in the appended data, but the actual photo attachments are left behind.

For example:

If report number 18-0807-1000 originally had 7 photo attachments in the repeat table (accessible and visible from the hosted Feature Layer's "Show Related Records" in AGOL Map Viewer, as seen in screenshot below)...

...the new Feature Layer with the appended data (exported from the original survey as a FGDB then appended in ArcGIS Pro) would have the same number of records in the related photo repeat table, but no "Photos and Files" to view (as illustrated in screenshot below).

The way it was explained to me:

"In short, as of now, there is no easy way to append features with repeats to a new feature class/service, regardless of a schema change. This is because repeats consist of a relationship with a related table based on a global ID, and the Append tool regenerates the global ID for each record when it is appended to the target layer. Essentially, when the new global ID is generated, the relationship to the repeat table is broken."

Makes sense, I suppose.

As of now, my understanding is that ENH-000113718 (Allow Workflow or ability for Survey123 to load, or append all records from a previous survey that included Repeats, i.e relationships in the Hosted Feature Layer) is currently marked as Not in Product Plan.

This is the suggested workaround: https://community.esri.com/groups/survey123/blog/2018/07/17/how-to-turn-supportsapplyeditswithgloablids-on-your-feature-layers

The second enhancement, ENH-000113717 (Add ability or workflow that would allow Append operations to preserve Global ID for hosted feature layers with relationships), is In Product Plan.

If anyone has luck with this workaround, please share your scenario and let us know how it worked out. :-]

Re: Combining two feature layers for Survey 123

GaryBowles1 — Tue, 20 Nov 2018 15:52:59 GMT

We ran into the issue of attachments not loading into new SDE datasets that we created to deal with 3D data. We developed a work-around for this that was fairly complex but it worked. I have attached a powerpoint from a presentation i gave about this. If you want more info, or need some clarification email me and we can see if this will work for you. There may need to be some modifications made for the repeat tables but I think it should work.

good luck,

--gary

Re: Combining two feature layers for Survey 123

ShanaGail1 — Tue, 20 Nov 2018 22:12:10 GMT

Hi Gretchen,

Thanks you for the update. We ended up reattaching the photos to the HFL by writing a Python script, with some pre-work using FME. Would that be an option for your organization?

Shana

Re: Combining two feature layers for Survey 123

GretchenFolse — Fri, 30 Nov 2018 01:06:53 GMT

Just a quick (ha!) update for everyone...

After a conversation with our awesome support rep Morgan, I’m understanding more about how all this fits together and what's actually happening (or not happening) in the Append process we’re using to try to migrate our survey data to its updated/republished form schema…

Y'all bear with my pedantic approach to explaining this... I'm usually just the Excel data geek who makes the Survey123 forms look pretty and do what the users want them to do, so I'm still a newbie at this side of ArcGIS. :-]

Each of our surveyPoint records has a GlobalID that is then related to each of its attachments in the photo_attachments_repeat table….

…each of which has its own GlobalID, and ties back to the surveyPoint with the referenced ParentGlobalID.
And this relationship is passed on via the REL_GLOBALID to the actual photo attachment files in the photo_attachments_repeat_ATTACH table…

So it looks like the steps in the workaround for that first enhancement ENH-000113718 (Allow Workflow or ability for Survey123 to load, or append all records from a previous survey that included Repeats, i.e relationships in the Hosted Feature Layer) is probably unnecessary for us—each of our records already has a GlobalID, ParentGlobalID, and a REL_GLOBALID to tie all the pieces together.

And our GlobalIDs for the "photo_attachments_repeat" table is obviously making it through the Append process intact because the resulting HFL shows the same number of “placeholders” for the photos as there are photo attachments in the originally collected survey data.

The problem seems like it might be with preserving our "REL_GLOBALID" where the relationship between the "photo_attachments_repeat" and the "photo_attachments_repeat_ATTACH" tables is just not being maintained through the Append process. Which might be where ENH-000113717 (Add ability or workflow that would allow Append operations to preserve Global ID for hosted feature layers with relationships) comes in. And it sounds like this enhancement is in the product plan, so we may just hold off on making the more detrimental schema adjustments to our form for now.

Shana Gail‌ - Shana, thanks for the update! We've actually been looking into purchasing a license for FME Desktop for ESRI to work with our engineers on another project we have going on with pump stations, so this could very well be our next step. I might reach out to you for some feedback once we have it installed and we've familiarized ourselves with the interface.

Gary Bowles‌ - Gary, thanks for the attachment and the info! I'm not fluent in Python (see also: basically clueless), but I'm sure I'll have to dive in and start a self-taught crash course soon if the enhancement request for preserving the GlobalIDs for relationships doesn't pan out.

Again, apologies for the stream of consciousness dissertation and if I'm over-explaining things; I tend to ramble when I'm over-caffeinated, and I'm kinda using this thread as my lab notes to jog my brain as I go through the troubleshooting and testing processes.

And if my notes can help someone else or paints a clearer picture to the Development team of the problem -- yay!