Error Uploading Linestring Dataframe from Databricks to ArcGIS Enterprise Portal: "No supported append format found."

Vegove

Background:

We've collected road data using an API and imported it into Databricks with the intention of displaying it on maps in ArcGIS Enterprise Portal. However, I'm encountering a problem when trying to upload a dataframe containing linestrings to the feature server in ArcGIS Enterprise Portal.

Issue:

When I attempt to save the dataframe from Databricks to the feature server, I receive the following error:

"java.lang.Exception: No supported append format found"

I'm not using the append command; I'm simply trying to save the dataframe. Initially, I thought the issue might be related to the data format, but the data is structured correctly with:

• A geometry shape column
• Spatial reference
• An OBJECTID with the datatype INT

What's puzzling is that I can upload a dataframe with point geometries to ArcGIS Enterprise Server without any problems, but linestrings result in the error.

Troubleshooting Steps Taken:

1. Data Format Verification:

   • I read an existing dataset from the feature server into Databricks.
   • Attempted to save it back to the server after renaming it.
   • Result: Encountered the same error.
   • Conclusion: Since the original dataset works fine on the server, this suggests the issue isn't with the data format.

2. Version Comparison:

   • Between January and April 2024, I successfully used GeoAnalytics version 1.3.0 with a script to write a linestring dataset to the server.
   • Currently, using the same process, I receive the error.
   • Note: We're now using GeoAnalytics version 1.4.

Current Environment:

• Databricks Cluster:

  • GeoAnalytics version 1.4
  • Spark version 13.3 LTS (Apache Spark 3.4.1, Scala 2.12)

• ArcGIS Enterprise Portal:

  • Version 10.9.1
  • Recently moved to a different server in preparation for an upgrade.

Additional Observations:

• I can successfully save data to a feature server in ArcGIS Online without any issues.

Questions:

1. Potential Causes and Solutions:

   • Does anyone have suggestions on what might be causing this issue?
   • Are there alternative approaches I could try to resolve it?

2. Compatibility Documentation:

   • Where can I find documentation regarding which versions of ArcGIS Enterprise Portal are compatible with GeoAnalytics 1.4?

3. Possible Factors:

   • Could the problem be related to a specific standard not supported when writing from Databricks?
   • Might firewall restrictions or server configurations be causing this issue?

4. Best Contact at Esri:

   • Which source is the best contact to use at Esri to get an answer to this problem?

I'm open to all suggestions and would greatly appreciate any insights.

 

 

ArtemisFili

Hey @Vegove 

We are able to reproduce this at our end. The issue is specific to the upload format being chosen when writing to the service, it is not related to the Databricks environment or firewall restrictions.

Here is an alternative approach that can resolve this: 

You can set the appendFormat to "featureCollection" and coalesceAppends to "False". The following code snippet can be used:

spark.conf.set("geoanalytics.sql.featureService.appendFormat", "featureCollection")
spark.conf.set("geoanalytics.sql.featureService.coalesceAppends", "False")

Can you please try it and let us know if it resolves the issue at your end?

-Artemis