Bulk upload data to survey 123

Question

Is it possible to upload existing survey data into a Survey 123 survey? We have data from an old survey (not Survey 123) that we wish to store/ combine with data in the Survey 123 survey.

This question was asked in 2018 but I am hoping the answer might be different now.

JamiDennis · Answer

It's been a few years since this was asked, but since I couldn't find good documentation on this, thought I'd share my solution in case someone like me is seeking a way to bulk add records to a Survey123 feature class. ArcPro with append would work but I opted for a Python script and appending an excel file.

A few notes: I had a few issues with some entries that needed to be truncated because they exceeded the 255 or 1000 string limit, so this accounts for that. I also had issues with null values when I first tried using a CSV file (python kept putting "nan" for null) but switching to an XLSX file removed that issue.

With that preamble - here is the Python code I used to bulk add records from an XLSX file to a Survey123 feature class:

This assumes you have arcgis, pandas, and numpy installed

from arcgis.gis import GIS
from arcgis.features import FeatureLayer
from arcgis.geocoding import geocode
import pandas as pd
import numpy as np
import arcgis.env as env

# Connect to your GIS environment
gis=GIS("https://www.arcgis.com","YOUR USERNAME HERE","PASSWORD")

# Set the active GIS environment
env.active_gis=gis

# Access the feature layer
feature_layer_url="URL TO THE HOSTED FEATURE CLASS"
feature_layer=FeatureLayer(feature_layer_url)

# Load your Excel data
excel_file="EXCEL XLSX FILE WITH FULL PATH"
df=pd.read_excel(excel_file)

# Define the maximum lengths
max_lengths = {
    'organization_name': 255,
    'category': 255,
    'Other_category': 255,
    'program_name': 255,
    'program_details': 1000,
    'eligibility': 1000,
    'website_url': 255,
    'supplementary_website_url': 255,
    'operating_hours': 1000,
    'location': 255,
    'contact_name': 255,
    'contact_email': 255,
    'phone1': 255,
    'extension1': None,  # Assuming extension fields are numeric
    'phone_type1': 255,
    'other_phone1': 255,
    'phone1_details': 255
}

# Function to truncate fields based on max length
def truncate_fields(df, max_lengths):
    for field, max_len in max_lengths.items():
        if max_len is not None:
            if field in df.columns:
                df[field] = df[field].astype(str).apply(lambda x: x[:max_len])
    return df

# Clean and convert data
df = df.fillna('')
df = df.apply(lambda col: col.map(lambda x: None if pd.isna(x) else x))

# Apply truncation
df = truncate_fields(df, max_lengths)

# Convert DataFrame to dictionary
df_dict = df.to_dict(orient="records")

# Function to geocode an address
def geocode_address(address):
    if address:  # Check if address is not empty
        results = geocode(address)
        if results:
            location = results[0]['location']
            return location['x'], location['y']
    return None, None

# Create features from the Excel data
features = []
for record in df_dict:
    x, y = geocode_address(record.get("location"))
    feature = {
        "attributes": record,
        "geometry": {
            "x": x,
            "y": y,
            "spatialReference": {"wkid": 4326} if x is not None and y is not None else None
        } if x is not None and y is not None else None
    }
    features.append(feature)

# Add features to the feature layer
if features:
    response = feature_layer.edit_features(adds=features)
    print(response)
else:
    print("No valid features to add.")

dougbrowning · Answer

Use ArcPro and then the Append tool.  In pro you can use services in all the tools so just add the hosted layer and Append.  Super easy.

I would not use Load Data as it does not handle transformations and such.

Alena · Answer

I came across this, not too long ago. I haven't tried it yet. Maybe, the procedure could help.

https://support.esri.com/en/technical-article/000024002