Migrating data from the Survey Feature Service to an Enterprise Geodatabase

65340
227
12-14-2016 12:23 PM
JamesTedrick
Esri Esteemed Contributor
38 227 65.3K

As you may know Survey123 for ArcGIS creates its own feature service when creating a survey. This exists in either ArcGIS Online's or your Portal's Hosted Data. From there, the data can be downloaded as a File Geodatabase easily. This makes it possible to import the data into a geodatabase, though this works best as a one-time import. To help synchronize an enterprise geodatabase with an on-going survey, we're introducing a Python Script and ArcGIS Toolbox Tool, available on ArcGIS Online. You may choose to use this as a one-type migration or as part of an ongoing data synchronization routine.

This script does the following:

  • Log in to ArcGIS Online and download the survey database
  • If needed, create the tables in the enterprise geodatabase to store the data
  • If applicable, filter out records that have already been synchronized over
  • Copy over the data from the database maintaining attachments and relationships

This was authored with ArcGIS Desktop 10.4, but will work with ArcGIS Desktop 10.3 if the pytz python module is installed.

Parameters used by the tool

The following parameters are needed:

  • SDE Connection File | File Geodatabase: Either a .sde file that connects to the destination database or a File Geodatabase
  • Table Prefix: The tables are created will have the prefix applied before the name of the table (i.e., if the prefix is 's123', a table in the service with the name 'myform' will be imported as 's123myform'). This is to prevent the chance of accidentally overwriting tables of the same name. This needs to be unique per survey.
  • Feature Service URL: the url to the Feature Service endpoint. This is the endpoint for the Service, not an individual layer (i.e., it will end with '/FeatureServer' with no numbers)
  • Time Zone: Dates in ArcGIS Online are stored in UTC time; python processes by default without a time zone. The time zone parameter is needed to calculate the difference. The time zones are specificed in this list
  • Portal: either 'https://www.arcgis.com' for ArcGIS Online or the Portal's url
  • Username: your username for ArcGIS Online / Portal
  • Password: your password for ArcGIS Online / Portal. If you don't supply it, you'll be prompted by the script after it starts

If you run the geoprocessing tool in ArcMap and are already signed into to your ArcGIS organization, the script will use that information instead of the Portal, Username and Password parameters.

How to use the script

There are three ways to use the script:

  • ArcGIS Script tool: This provides a graphical interface to the tool using ArcGIS Desktop. If you have an enterprise login, first sign into ArcGIS Online or your Portal for ArcGIS, within ArcGIS Desktop.
  • Command line: Invoked with parameters specified. The usage in this case is:
    python syncSurvey.py <SDE Connection File | File Geodatabase> <Table Prefix> <Feature Service Url> <Time Zone> <Portal> <Username> [Password]
  • Using a config file: The parameters above can be stored in a text file and read by the script, making automated syncronization jobs easier. To run in this mode, use the following syntax:
    python syncSurvey.py CONFIG <path to config file> <section name>.
    The config file has the following format:

[SECTION_NAME]

portal: https://www.arcgis.com

username:

password:

service_url:

sde_conn: <The path to a .sde file that connects to the destination DB OR a File GDB>

prefix: s123

timezone: US/Pacific

227 Comments
PhilipWilson-GIS
New Contributor III

Great, thanks James for the above info, this will be very useful.

DanielCruz1
New Contributor III

Hi, I am getting an error message with the timezone input. I tried "US/Arizona", "UTC-7", "Arizona", "UTC-07", "MST". The script game the same error, can someone please help me with this? I am trying to run the script from the command line.Thanks

JamesTedrick
Esri Esteemed Contributor

Hi Daniel, 

What is the error message being reported?  It looks like the authoritative tz entry should be "America/Phoenix"

DanielCruz1
New Contributor III

It is not taking any of my inputs on time zone, The tool works if I use "UTC" but does not work with any of the other inputs.

JamesTedrick
Esri Esteemed Contributor

Can you provide a screenshot or additional information?

DanielCruz1
New Contributor III

JamesTedrick
Esri Esteemed Contributor

Looks like the issue is capitalization.  The time zone code should be America/Phoenix (title-cased).  I noticed the link to the time zone list was mangled and have updated the post above to get you to correct listing.

DanielCruz1
New Contributor III

Yes it worked, but the time and date was exactly the same for all records. "12/20/2016 9:39:24 AM" . Any idea what happened? Thanks

JamesTedrick
Esri Esteemed Contributor

That sounds correct for the 'SYS_TRANSFER_DATE' field - it's a timestamp of when the synchronization took place.  It's recorded so that on future runs of the script, it can exclude records that have already been copied over- the value is read from the feature class and any surveys with a created date before then are excluded from the copy over.

BrianCollins
Occasional Contributor

JTedrick-esristaff‌ do you foresee this script synchronization being a long-term solution or a stop gap until there is native connectivity between Survey123 and feature services we host on our own ArcGIS Server and Enterprise Database environment?

JamesTedrick
Esri Esteemed Contributor

This may be a long-term solution for some orgs if an administrator wanted to have a level of separation between the user touched data interface and their system of record (if, for example, IT policies prohibited a data editing connection being exposed beyond the firewall).  It can also be a stopgap until a connection to a feature service connected to an enterprise database is available; we have provided some instructions with doing so with version 1.10 which was released yesterday- take a look at How To: Create Survey123 for ArcGIS surveys that connect to non-federated ArcGIS Server services  for instructions.

DanielCruz1
New Contributor III

We are planning to upgrade to enterprise 10.5 in the near future. Also we will install and configure portal for GIS so hopefully we will be able to host our own feature classes inside ArcGIS enterprise.

JoeFlannery
Occasional Contributor III

James:

 

This is a fantastic script – thank you for sharing with the user community.

 

I have successfully used your script to populate an on-prem FGDB from the ArcGIS Online hosted feature service.  I used Survey123 to create new survey records and, when run again, the script added/appended those new records to the on-prem FGDB.

 

Then, via ArcGIS Online, I added the Survey123 hosted feature service to a map and moved two points, thus altering the geometry and the value in the “EditDate” field.  I ran your script again and the featureclass in the FGDB was not updated to reflect the moved points.  Via ArcGIS Online, I then altered the values in some attribute fields of existing records, ran your script again and the FGDB was not updated with the attribute changes.

 

So, if I understand this blog and my test results, your script will only bring down new survey records if the “CreationDate” is newer than the current “SYS_TRANSFER_DATE” and not update/overwrite existing records if the “EditDate” reflects a change in geometry or attribute values.

 

Might you consider modifying your script to update edited records from the hosted feature service in ArcGIS Online to the on-prem FGDB?

 

  • “Append” new records if the CreationDate is newer than the current on-prem SYS_TRANSFER_DATE
  • “Overwrite” existing records if the EditDate is newer than the current on-prem EditDate

 

 

Thank you again for sharing your script.

Sincerely,

Joe

JoeFlannery
Occasional Contributor III

Note: Our ArcGIS Online subscription uses enterprise credentials to authenticate named users. In order to get James' stand-alone command line code + script to work, I needed to open ArcMap and sign into AGOL as a named user then run the command line script.  Also, to get not only new records from Survey123 additions, but also get any attribute or geometry modifications, I wrote a python script to rename the existing on-prem FGDB, create a new FGDB with the proper name and then populate the new FGDB with S123 features using James’ script.

StaceyDudenhoeffer
New Contributor II

This script is wonderful James!  With no experience in python scripting, I have been successful at migrating the Survey123 data from a hosted feature class in AGOL to my enterprise GDB using your instructions. 

I want to take this to the next level for our industry.  Once we have this data in our enterprise GDB, I have Registered my Data as Versioned.  Then I publish the data to ArcGIS for Server 10.3.1 with Edit, Sync and Query capabilities on the feature layer.  Then I am consuming that feature service in a web map with editing capabilities. My ultimate goal is to use the Collector App on and iOS device and take a map offline, edit the offline data, then Sync the data back into my enterprise GDB. 

I discovered a bug in Collector 10.4.2.  When I take this map offline, if I have two GUID fields in my FC (Global ID and Row ID), I cannot edit the points in an offline map.  As a test, I removed the Row ID field and my map works perfectly offline. 

My question is...why is the Row ID field necessary?  I believe I read in a post somewhere that this field was added to pick up attachments from a repeat question.  If I do not have repeat questions in my survey, is the Row ID field necessary?  If not, how can I modify the python script to not use Row ID?

JamesTedrick
Esri Esteemed Contributor

Hi Stacey,

Before version 1.10, the parent-repeat relationships were linked using the RowID column.  If your surveys don't have repeats (or, for the purposes of Collector, you're not using the repeats), you can easily hide the RowID field when publishing the service by hiding the RowID field.  

With version 1.10, we now support using the GlobalID as the field that the parent-repeat is based off of.  The script still creates a rowID field when there are attachments because the normal behavior of globalIDs is to be auto-generated, not set when copying over data; the script can't copy over the existing globalID info and needs a rowID as a place to store the previous value so that a lookup dictionary can be created.  RowID for used in this case can be hidden afterward as it doesn't participate in the attachment relationship.

GRSMGIS
New Contributor

Super Cool! Works flawless, and repeatedly, when syncing to a FGDB. Unfortunately, this script does not work when attempting to sync for a second (or any other time other than the first) time against an SDE database. When running this for a SECOND TIME (After the first successful sync and table creation) against a SQL2012 SDE 10.3.1 database, it returns

Setting Timestamp on GRSM_SASQUATCH_SRV
 -Creating Tables
 -Creating Domains
 -Creating Feature Classes & Tables
======================
FAIL: Making Tables
exception:
ERROR 000258: Output Database Connections\RESTON SASQUATCH.sde\GRSM_SASQUATCH_SRV_EVENTS_GRSM_SASQUATCH_SRV already exists
Failed to execute (CreateFeatureclass).
['ERROR 000258: Output Database Connections\\RESTON SASQUATCH.sde\\GRSM_SASQUATCH_SRV_EVENTS_GRSM_SASQUATCH_SRV already exists\nFailed to execute (CreateFeatureclass).\n']
<class 'arcgisscripting.ExecuteError'>
567
----------------------
arcpy messages:
ERROR 000258: Output Database Connections\RESTON SASQUATCH.sde\GRSM_SASQUATCH_SRV_EVENTS_GRSM_SASQUATCH_SRV already exists
Failed to execute (CreateFeatureclass).

According to the documentation...should not be the case? It should detect if the table exists and just append to it?

JamesTedrick
Esri Esteemed Contributor

Yes, it should be detecting the existing tables (there's a function that searches for the tables).  Can you verify that the table names in the database match the names as they appear in the script?  I'm wondering if there might be an issue over schema/username (i.e., GIS.GRSM_SASQUATCH_SRV_EVENT - this should be found by the search pattern) or capitalization (GRSM vs grsm).

ThomasColson
MVP Frequent Contributor

I think I got past that. Deleted everything and started over...and seems to be working now on 1st and subsequent syncs. The problem now is, when I run the tool from Arc Catalog a SECOND time, I get an in-memory workspace error. Looking at the py code, it appears that temporary data written to  in-memory is not cleaned up. 

arcpy messages:
Failed to execute. Parameters are not valid.
ERROR 000725: Output Table: Dataset in_memory\stat_EVENTS_GRSM_SASQUATCH_SRV already exists.
Failed to execute (Statistics).‍‍‍‍

If I close and restart Arc Catalog, I can then run the tool again. Wondering if the py should be modified to add an in-memory cleaner, or add datetime stamp to the dataset name in memory?

Testing the proposed solution...adding

arcpy.Delete_management("in_memory")

at the end of the script now allows it to be run multiple times during one ArcCatalog session. 

JamesTedrick
Esri Esteemed Contributor

Thanks for the catch- I'll add a Delete statement to clear up the stat tables (all other points where I create an in_memory table I already had it clean up at end).

ThomasColson
MVP Frequent Contributor

This might fall into the range of enhancement request, but despite my tinkering with the PY code (removing asking for the id and pw), running this in Arc Cat/Map while logged in using an enterprise account....doesn't work with numerous PY exceptions. As many organizations are moving toward that authentication model.....

JamesTedrick
Esri Esteemed Contributor

Hi Thomas,

Can you provide the errors you are receiving?  The script should be inheriting you're login credentials if you're already signed in- I set it up that way to specifically accommodate enterprise logins.

ThomasColson
MVP Frequent Contributor

I stopped tinkering with the PY and as you say...it works...when an org user is logged on. But...back to the tinkering...this may be more of an enhancement/not possible request...where I'm really trying to go with this it to automate it so I can run it nightly under task scheduler. I'm not sure it's possible to attach an Active Directory auth certificate, say, from a service account, in place of your named user part of the login? I'm finding all sorts of neat uses for your script. 

FRAP_TiffanyMeyer
New Contributor

Could this script work for syncing data from an AGOL feature service to desktop or SDE server also?  I actually tried it, but got an error.

ThomasColson
MVP Frequent Contributor

Yes, it should, I've gotten it to work that way.

ThomasColson
MVP Frequent Contributor

When using a survey that has repeat questions, which, in turn, results in survey answers being stored in database relationships, invoking SyncSurvey.py returns the following error:

C:\PY_SCRIPTS\SURV_SYNC>python syncSurvey.py SASQUATCH.gdb GRSM_HWA_MON_SRV_TEST https://services1.arcgis.com/orgid/ArcGIS/rest/services/service_someguid/FeatureServer US/Eastern https://www.arcgis.com
 -Getting Replica
{"replicaName":"","replicaID":"","submissionTime":1486556009510,"lastUpdatedTime":1486556013337,"status":"ExportingData","resultUrl":""}
 -Check 1: ExportingData
{"replicaName":"","replicaID":"","submissionTime":1486556009510,"lastUpdatedTime":1486556016447,"status":"Completed","resultUrl":"https://services1.arcgis.com/orgid/ArcGIS/rest/services/service_someguid/FeatureServer/replicafiles/4faa340f83ea42829aa9ed0696ebc7b9.zi
p"}
 -Check 2: Completed
 -Temporary Directory: c:\users\user\appdata\local\temp\1\tmplwcpbq
f1122fc78a194a749233fb8d573e8689.gdb
 -Filtering records to new set
 -CreationDate > date '2017-02-08 12:13:20'
 featureclass
 -Adding Syncronization Time
 -Creating Tables
 -Creating Domains
 -
 -
 -
 -
 -
 -
 -
 -
 -
 -Creating Feature Classes & Tables
 -Created GRSM_HWA_MON_SRV_TEST_GRSM_HWA_Mon
 -Created GRSM_HWA_MON_SRV_TEST_IndividualTree
 -Created GRSM_HWA_MON_SRV_TEST_BiocontrolSpeciesData
 -Creating Relationships
======================
FAIL: Making Tables
exception:
Failed to execute. Parameters are not valid.
ERROR 000800: The value is not a member of globalid.
Failed to execute (CreateRelationshipClass).
['Failed to execute. Parameters are not valid.\nERROR 000800: The value is not a member of globalid.\nFailed to execute (CreateRelationshipClass).\n']
<class 'arcgisscripting.ExecuteError'>
562
----------------------
arcpy messages:
Failed to execute. Parameters are not valid.
ERROR 000800: The value is not a member of globalid.
Failed to execute (CreateRelationshipClass).
======================
C:\PY_SCRIPTS\SURV_SYNC>
C:\PY_SCRIPTS\SURV_SYNC>
C:\PY_SCRIPTS\SURV_SYNC>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍
JamesTedrick
Esri Esteemed Contributor

It looks like the script is being affected by a known issue with the create relationship tool (NIM082537).  The tables are created, but the parent-repeat relationship isn't.  Subsequent runs of the script will still copy the data over, as the tables are already present.

Another workaround would be to not use the GlobalID as the relationship key- see Publish your survey—Survey123 for ArcGIS | ArcGIS for doing this; it's a option you would set at time of publishing.  There is an impact on using the feature service in other applications, like Collector.

ThomasColson
MVP Frequent Contributor

That did the trick, thanks!

Regarding the relationships for my application, this is not an issue as I'm pulling this into SQL and using the data outside of Esri software. 

ThomasColson
MVP Frequent Contributor

Enhancement request:

We have a rather onerous table/object-naming convention for our SQL SOP. I've been trying, unsuccessfully, to edit the py to "bake" in a user-specified table(s) name, and not the {user supplied name}_{name of survey}. Any chance you could indicate where, in the py, one could accomplish that? 

TL2
by
Occasional Contributor III

Am I correct that we need to Enable Sync from within the edit tab of the Survey's hosted feature service?  When I do this it breaks all of my attachments to the survey and I get this error when trying to submit their after. 

ThomasColson
MVP Frequent Contributor

More "Feature Class Already Exists" errors. 


So I've been getting a lot of this, and getting really frustrated:

ERROR 000258: Output HWA.gdb\HWA_MON_SRV_GRSM_HWA_Mon already exists
Failed to execute (CreateFeatureclass).

frustrated enough to run the py, 1 line at a time, to see what I was doing wrong. Near the end of the whole thing, a line asks the length of the string that is holding the table names, and I found that it was always returning "0", which, would trigger the create tables part of this, hence the error. 

So:

>>> prefix='HWA_MON_SRV'
>>> arcpy.env.workspace = 'HWA.gdb'
>>> originalWorkspace = arcpy.env.workspace
>>> wildcard = '*{0}*'.format(prefix) if prefix != '' else '*'
>>> featureClasses = arcpy.ListFeatureClasses(wildcard)
>>> tables = arcpy.ListTables(wildcard)
>>> allTables = featureClasses
>>> allTables.extend(tables)
>>> outTables = []
>>> for t in allTables:
 tableName = t.split('.')[-1]
 nameParts = tableName.split('_')
 if '__ATTACH' not in t:
 if nameParts[0] == prefix or prefix == '':
 outTables.append(t)
>>> arcpy.env.workspace = originalWorkspace
>>> outTables
[]
>>> wildcard
'*HWA_MON_SRV*'
>>> nameParts
[u'HWA', u'MON', u'SRV', u'BiocontrolSpeciesData']‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Sure enough, outTables is NULL. 


There's a tablesplit thing going on here, based on the underscore "_".

So what if I DO NOT include an underscore in my prefix name? 

>>> prefix='HwaMonSrv'
>>> arcpy.env.workspace = 'HWA.gdb'
>>> originalWorkspace = arcpy.env.workspace
>>> wildcard = '*{0}*'.format(prefix) if prefix != '' else '*'
>>> featureClasses = arcpy.ListFeatureClasses(wildcard)
>>> tables = arcpy.ListTables(wildcard)
>>> allTables = featureClasses
>>> allTables.extend(tables)
>>> outTables = []
>>> for t in allTables:
 tableName = t.split('.')[-1]
 nameParts = tableName.split('_')
 if '__ATTACH' not in t:
 if nameParts[0] == prefix or prefix == '':
 outTables.append(t)

>>> outTables
[u'HwaMonSrv_GRSM_HWA_Mon', u'HwaMonSrv_IndividualTree', u'HwaMonSrv_BiocontrolSpeciesData']
>>> nameParts
[u'HwaMonSrv', u'BiocontrolSpeciesData']
>>> wildcard
'*HwaMonSrv*'

And now the length of the variable returns "3". 

Take home message: 

Don't include underscores in your prefix string!

JamesTedrick
Esri Esteemed Contributor

Hi Thomas,

My apologies - when I developed the pattern, I was intending the prefix string to be relatively short; I'm logging this as an issue.

deleted-user-st4HsVxccxgs
New Contributor III

This may be an issue with the date attribute, but my records keep duplicating in this fashion:

If I have 1 original record, 1 record is populated in my SDE feature class upon first run.
I submit 1 additional survey record, giving me 2 original records.
Upon 2nd run, my SDE feature class will have 3 records (should only be 2).
Upon 3rd run, my SDE feature class will have 5 records (should only be 2).
Upon 4th run, my SDE feature class will have 7 records (should only be 2).

Essentially, it adds my original records fresh each time as new records.

We are using timezone: America/Chicago in the config, but the Sys_Transfer_Date still seems to be off. Not sure if this would contribute or not. 

TroyTreat
New Contributor II

Hi James or anyone that knows, I'm getting this Unauthorized error at this line of your code in getReplica Function.  I'm not sure why I'm getting it because a few lines above I set a print statement that creates what I believe is the proper URL.  I can copy and paste that URL into a browser and get the json location of what I assume is the zip file of the gdb.  Thoughts?

JamesTedrick
Esri Esteemed Contributor

Hi,

Can you describe the setup for the item- is the user of the item the owner for it?  If not, is export operations for other users set up?

I'm assuming the script hasn't been running for an hour or longer (you could try increasing the expiration parameter of the user in the login process if it is).

TroyTreat
New Contributor II

Hi James, I am only able to make more progress using a gdb as the output as opposed to SDE.  That being said I can get farther in getReplica function but its dieing at this area below.  The zip file does get created but its empty

JamesTedrick
Esri Esteemed Contributor

I assume it's not empty if you attempt to export the FGDB via the ArcGIS Online interface?  This is sounding like an issue with this dataset.

TroyTreat
New Contributor II

Hi James, there was a single feature in the feature layer in ArcGIS Online, but I collected 2 more to prove this is not the issue.  I queried the FeatureServiceURL with token for all features and received count of 3 which is accurate.  I really need to get this to work directly with SDE and as much trouble as I'm having just to get it to work with a gdb (Having to use python window in arcmap which is not preferred ) I'm stuck.  Is there a faster way I can get support on this to get this issue resolved?  Ticket with ESRI? 

TroyTreat
New Contributor II

Hi James,

I was able to get this moving forward by using ArcREST to pull down the gdb from the cloud.  I then used the majority of your script (no token and replication needed) to do the Appending, Relationships, etc.  I'm able to insert records successfully which is great.  Only thing missing is the Attachments as it appears the script did not bring over the attachment/relationship pieces to SDE (However it did successfully create the SDE featureclass).  I'm stepping through the code to see why but if you have any further suggestions, I'd love to hear them.  Thanks again James

by Anonymous User
Not applicable

Hi Thomas and JTedrick-esristaff,

What are the requirements for using this tool to get data from a typical Esri-hosted feature service (non-Survey123)? Need the feature service have a CreatedDate field?

I've tried publishing a feature layer (from a versioned SDE workspace) to an Esri-hosted feature service, making some edits (delete, add, update geometry and attributes), then running this tool. Here is the error I got:

If I can get this working I'll be a hero. I had tried rolling my own solution to this problem but couldn't get anything to work consistently.

Thanks,

Micah

JamesTedrick
Esri Esteemed Contributor

Hi Micah,

The script is expecting editor tracking to be turned on and the field be called CreationDate - that's used to conduct the synchronization process.  Aside from that, it should work against any other sync-enabled Feature Service (there isn't a Survey123 Feature Service, just a Feature Service that Survey123 writes to).

João_Mendes
New Contributor II

Hi, James,

You can illustrate how I can configure a configuration file with parameters to automate easier synchronization tasks. I just need to see an exemple.

JamesTedrick
Esri Esteemed Contributor

Take a look at SyncSurvey/config file template.txt at master · tedrick/SyncSurvey · GitHub ; a copy of it should have been in the zip file.

MariaNieves
New Contributor III

Hi,

I am trying to use this tool and haven't been successful so far. I have tried with both FGDB and SDE and get the same error. 

Any ideas? I am running the script on ArcCatalog 10.5. 

Thanks 

JamesTedrick
Esri Esteemed Contributor

Hi Maria- there is a point of brittleness in the script in that it expects the service to have Editor Tracking enabled and the field that marks when the record was created to be called 'CreationDate' (see https://community.esri.com/groups/survey123/blog/2016/12/14/migrating-data-from-the-survey-feature-s...  ).  Take a look at your service's fields and see if that's the case; you can modify the field name in the script (either search/replace or of to line 207,208,211)

MariaNieves
New Contributor III

Thanks James. I was able to get passed that error but now I am running into this other error. I am trying to sync to an empty FGDB

FAIL: Updating Tables

exception:

FieldMap: Error in adding input field to field map

[u'FieldMap: Error in adding input field to field map']

<type 'exceptions.RuntimeError'>

572

Also, my goal is to sync my survey123 data to an oracle SDE GDB. There is a 30 character limit for Oracle feature classes and table names and the SurveyGDB names are too long. Would I have to make significant amount of changes to the script to get this script to work with Oracle?

Thanks

Maria  

JamesTedrick
Esri Esteemed Contributor

Hi Maria,

Given the issue with the destination database, the error is occurring at a place where you may want to change the code of the script to fit your particular situation.  The Field Map objects provide the script information about which field in the surveyFC corresponds to the field in the destination FC.  Rather than automatically create them, you might want to manually specify them in the code.

MariaNieves
New Contributor III

Thanks. I was able to get the script to run successfully with the same survey but published in AGO and syncing to a FGDB. I noticed the editor tracking fields created when publishing the survey on AGO are different.than the ones created when publishing to Portal for ArcGIS with an ArcGIS Data Store. This is a little challenging considering that I have no way to control that. So I think my issues are related to publishing my survey in Portal for ArcGIS and trying to sync data from the ArcGIS Data Store. This is a requirement for us so I will try to change the python script to see if I can make it work.  

BretLian
New Contributor II

Hi James,

Thanks for sharing this script. We have been using it with success. I am currently running into an error on a feature service for Survey 123 that we are hosting in AGOL and would like to pull down into SDE. This is the error:

This error, 000112 : Domain does not exist. The domain you are attempting to use or manipulate does not exist within the workspace specified.

Are domains in SDE analogous to the list_name on the choices tab of the .xlsx spreadsheet the survey was created from? I duplicated the list_name options as domains in SDE, where the name and label columns in the spreadsheet matched the code and description columns for the corresponding domains.

Thank you,

Bret Lian

JamesTedrick
Esri Esteemed Contributor

Hi Bret,

Yes, in Survey123 a select_one question's possible values become a domain in the feature service when published.  In your list, I see a bit of weirdness - there's a choice list with the name of ";" (it looks like a mistake in the Appearance list)

About the Author
I work on the Survey123 team. Please note that I do not use the private messages feature in Esri Community; if you need to reach me, feel free to e-mail jtedrick@esri.com .