topic Submission Failure and Duplicate Records in ArcGIS Survey123 Questions

Submission Failure and Duplicate Records

PeterTimmers — Sun, 05 Nov 2017 23:52:45 GMT

We've had a number of people have the situation where they have gone to submit surveys and S123 says it has failed, they submit again, then we find duplicate records in the feature layer.

We have a feature service hosted on AWS servers and registered on AGOL.

It's not repeatable and I haven't had it happen on any of my tests, but we've had a number of people in different areas had it happen.

We currently rely on a scheduled task to look for duplicates (using S123's unique id field).

It may have to do with photo size and internet connectivity and we've reduced our photo size to reduce the risk.

Has anyone else experienced this?

Re: Submission Failure and Duplicate Records

Anonymous User — Mon, 06 Nov 2017 00:03:51 GMT

Hi Peter,

Short answer, yes, this issue has been experienced by other users and is reproducible with surveys with attachments when the network connection is unstable or slow. Currently the survey record is uploaded first, and then the attachments uploaded. If the attachments fail due to network connection loss, the survey record will already exist in the feature service but with no attachments. When the survey is sent again, it will create a duplicate with the attachments. To avoid this issue, surveys with attachments should be uploaded when the network connection is stable and fast and as you suggest using a smaller photo size is recommended.

We also recommend ensuring that all users have the latest version of the app installed, as we are constantly working on improving this issue and other app related bugs. We currently have a feature enhancement to resolve the upload failure bug, and hope to implement this is a future release.

Regards,

Phil.

Re: Submission Failure and Duplicate Records

JuanLuera3 — Tue, 05 Jun 2018 18:04:35 GMT

Hi Phil,

Would you be able to share either logged Bug's or Enhancement information? We could then track this with Esri Support Services, and for our own records. Thanks, Phil!

Juan

Re: Submission Failure and Duplicate Records

Anonymous User — Wed, 06 Jun 2018 00:41:06 GMT

Hi Juan,

Changes were implemented in version 2.6 and 2.7 of the app to improve the handling of uploading attachments and reduce the occurrence of duplicate survey entries if attachments fail. There was no specific external support bug or enhancement request for this issue, it was implemented as part of our continuous improvement and feature releases. Note that if you are using ArcGIS Online and the last version of the app, you will be using this new method for upload, however if you are using Portal it will depend on the Portal version and whether you have federated services. The feature requires the use uploadids capability to be set to true and supported by the Portal.

Phil.

Re: Submission Failure and Duplicate Records

PatriciaAlgura — Thu, 01 Oct 2020 03:22:17 GMT

Hi Phil,

Can this issue also occur to surveys with repeats? Was this issue ever fixed?

Pat

Re: Submission Failure and Duplicate Records

Anonymous User — Sun, 04 Oct 2020 22:53:09 GMT

Hi Pat,

Yes, this issue was fixed as described above, the process of how attachments are uploaded is the same for parent layer or repeats. The attachment is uploaded first, then the feature, and then they are related to each other. That way if there is a failure you will not end up with duplicate attachments to the same feature.

Are you seeing an issue? If so can you share more details about it and the workflow being used?

Regards,

Phil.

Re: Submission Failure and Duplicate Records

PatriciaAlgura — Mon, 05 Oct 2020 03:18:47 GMT

Hi Phil,

We had features with duplicate records (repeats) in them. Can I send you an email with the details?

Thanks,

Pat

Re: Submission Failure and Duplicate Records

epreskienis — Mon, 04 Oct 2021 13:29:46 GMT

Hello, I have the same problem as Patricia. Getting double records after submission. Im using MSflow to records new submissions from Survey123 form. Anyway to fix this? My form also has repeats.

Re: Submission Failure and Duplicate Records

Anonymous User — Tue, 05 Oct 2021 23:37:16 GMT

Hi @epreskienis,

Can you explain in more detail where you are seeing the double records? In the feature layer being used byt he Survey123 form, if so in the parent layer or related table? Or do you mean in the MSFlow you have created that is submitting data to another source?

Can you provide screenshots or more information to help troubleshoot this?

Regards,

Phil.

Re: Submission Failure and Duplicate Records

Fernando_LuisRodríguez_Valle — Mon, 08 Nov 2021 21:47:40 GMT

In our case, in the feature layer used by the Survey123 form, (in the main layer). We don´t use Msflow.

Users have the latest version application (Survey123) installed

Re: Submission Failure and Duplicate Records

Anonymous User — Mon, 08 Nov 2021 22:09:50 GMT

Hi @Fernando_LuisRodríguez_Valle,

Does this happen for all surveys, or just one? Are you using the Inbox and editing existing records, or creating new records? Do all records submitted get duplicated or just some?

Do the surveys have attachments?

Regards,

Phil.

Re: Submission Failure and Duplicate Records

Fernando_LuisRodríguez_Valle — Mon, 08 Nov 2021 22:29:47 GMT

Hi Philip

Does this happen for all surveys, or just one?

For the moment it is a general problem that happens intermittently (sometimes yes and sometimes not)

Are you using the Inbox and editing existing records, or creating new records?

Come about editing existing records and creating new records

Do all records submitted get duplicated or just some?

Only record with a Submission Failure

Do the surveys have attachments?

Yes we have a attachments (photographics) and unstable connection sometimes

*****************************************************

<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Logging started: 2021-11-08T13:48:54
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Application Name: ArcGISSurvey123
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Application Version: 3.13.234
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Application File Path: /private/var/containers/Bundle/Application/FF8B1433-4D0C-4040-A431-A0FF3DB0D48C/AppStudioApp.app/AppStudioApp
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Application Directory Path: /private/var/containers/Bundle/Application/FF8B1433-4D0C-4040-A431-A0FF3DB0D48C/AppStudioApp.app
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ AppFramework Version: 5.1.80
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Qt Version: 5.15.2
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Operating System: iOS 15.1
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Operating System Type: ios
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Operating System Version: 15.1
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Kernel Type: darwin
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Kernel Version: 21.1.0
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Build ABI: arm64-little_endian-lp64
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Current CPU Architecture: arm64
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Build CPU Architecture: arm64
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ SSL Library Version: Secure Transport, iOS 15.1
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Host name: Diego
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Domain name:
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Default locale: es_GT
<14>1 2021-11-08T19:48:54Z Diego ArcGISSurvey123 16492 - - $ Release Type: store
<12>1 2021-11-08T19:49:02Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="default"] stale focus object AppSwitch_QMLTYPE_411(0x2810c5550) , doing manual update
<15>1 2021-11-08T19:49:36Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="onStateChanged" filename="Portal.qml" line="404"] Portal: Application state changed: 2
<15>1 2021-11-08T19:49:37Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="onStateChanged" filename="Portal.qml" line="404"] Portal: Application state changed: 0
<15>1 2021-11-08T19:49:54Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="onStateChanged" filename="Portal.qml" line="404"] Portal: Application state changed: 2
<15>1 2021-11-08T19:49:55Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="onStateChanged" filename="Portal.qml" line="404"] Portal: Application state changed: 4
<15>1 2021-11-08T19:49:55Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="onRunningChanged" filename="Portal.qml" line="903"] Portal: Token expiry timer disabled
<15>1 2021-11-08T19:49:55Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="onRunningChanged" filename="Portal.qml" line="901"] Portal: Token expiry timer will trigger in 18.3075 minutes
<15>1 2021-11-08T19:49:55Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="reset" filename="Portal.qml" line="919"] Portal: Reset token expiry timer: lun nov. 8 14:09:13 2021 GMT-0600 minutes: 18.3075
<12>1 2021-11-08T19:49:58Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="default"] stale focus object QQuickTextInput(0x28100def0) , doing manual update
<12>1 2021-11-08T19:49:58Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="default" filename="XFormBarcodeScanView.qml" line="425"] qrc:/qml/XForms/XFormBarcodeScanView.qml:425:9: QML XFormText: Cannot anchor to an item that isn't a parent or sibling.
<12>1 2021-11-08T19:49:58Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="default" filename="XFormBarcodeScanView.qml" line="425"] qrc:/qml/XForms/XFormBarcodeScanView.qml:425:9: QML XFormText: Detected anchors on an item that is managed by a layout. This is undefined behavior; use Layout.alignment instead.
<12>1 2021-11-08T19:50:00Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="default"] stale focus object TextBoxButton_QMLTYPE_127(0x2810ba5d0) , doing manual update
<12>1 2021-11-08T19:50:00Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="default"] stale focus object StyledImageButton_QMLTYPE_34(0x2810e44d0) , doing manual update
<12>1 2021-11-08T19:50:01Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="default"] stale focus object StackView_QMLTYPE_312_QML_1231(0x281050fd0) , doing manual update
<15>1 2021-11-08T19:50:14Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="onStateChanged" filename="Portal.qml" line="404"] Portal: Application state changed: 2
<15>1 2021-11-08T19:50:15Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="onStateChanged" filename="Portal.qml" line="404"] Portal: Application state changed: 0
<15>1 2021-11-08T19:50:42Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="onStateChanged" filename="Portal.qml" line="404"] Portal: Application state changed: 2
<15>1 2021-11-08T19:50:42Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="onStateChanged" filename="Portal.qml" line="404"] Portal: Application state changed: 4
<15>1 2021-11-08T19:50:42Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="onRunningChanged" filename="Portal.qml" line="903"] Portal: Token expiry timer disabled
<15>1 2021-11-08T19:50:42Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="onRunningChanged" filename="Portal.qml" line="901"] Portal: Token expiry timer will trigger in 17.5189 minutes
<15>1 2021-11-08T19:50:42Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="reset" filename="Portal.qml" line="919"] Portal: Reset token expiry timer: lun nov. 8 14:09:13 2021 GMT-0600 minutes: 17.518866666666668
<15>1 2021-11-08T19:50:48Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="onStateChanged" filename="Portal.qml" line="404"] Portal: Application state changed: 2
<15>1 2021-11-08T19:50:50Z Diego ArcGISSurvey123 16492 - [AppStudio@11205 category="Portal" function="onStateChanged" filename="Portal.qml" line="404"] Portal: Application state changed: 0

************

Re: Submission Failure and Duplicate Records

Anonymous User — Tue, 09 Nov 2021 00:48:54 GMT

Hi @Fernando_LuisRodríguez_Valle,

Thanks for answering those questions. From the logs I don't see anything related to submission errors. From your explanation it sounds as if during the submissions, the network drops out, meaning some or all of the data has been sent for that survey, but the app has not recieved a reply to say it was successful, meaning it gets moved to Sent box. This would eventually cuase an error (timeout) and if you submit again, may cause a duplicate record, as a new globalid would be created. I assume the duplicate you see have different globalids?

The best way to ensure this does not happen, is always save surveys to the Outbox in areas of poor network connection, and only send them when the connection is solid and surveys can be sent without interuption. This is important when you have a lot of images or large attachments in your surveys, as they add to the time it takes to successfully send a survey.

Regards,

Phil.

Re: Submission Failure and Duplicate Records

Fernando_LuisRodríguez_Valle — Sun, 14 Nov 2021 15:53:55 GMT

Hi Phil,

Efectily , the duplicate register has a diferent global ID.

I share the screenshot

I compared the data and attachments of both records and they are identical but with different Global ID and therefore all the data was sent but the application not having confirmation issued the sending error as you had informed me.

In my opinion in these cases, it would be great if the application could double-check that all the data was sent before sending an error message on sending

Regards,