Overwrite Feature Service Error

[ClaireSullivan]

I am having a reoccurring error with the Spark DataFrameWriter on overwrite mode. I am attempting to overwrite a data layer with ~50,000 records. When I first run the cell, the Feature Service is truncated but no new records are loaded. If I re-run the same cell, the records are loaded to the Feature Service. Has anyone else experienced this error? 

[SBattersby]

I have not experienced this.  Does this happen with one specific dataset or with every dataset that you try? Do you receive an error at all from the first run?  If so, can you provide it?

And can you send a code example of what is/isn't working (redacting any credentials or feature service identifiers)?

Thank you,

Sarah.

[ClaireSullivan]

So far, I have only been using one dataset. There is no error, the run ends successfully but my dataset in AGOL is truncated instead of the new dataset being uploaded. When I run the cell again, the records are updated in AGOL. I am using this method to update my feature service: 

## upload feature service

featureServiceDf.write.format("feature-service")\

    .option("gis", "myGIS")\

    .mode("overwrite")\

    .option("serviceUrl", myURL)\

    .option("layerId", 0)\

    .save()

[SBattersby]

Hi Claire - we are working on reproducing this internally to see if we can figure out what might be happening.  Just for a few more details for our testing - is your ~50,000 record dataset points, lines, polygons?  If lines or polygons, are they really complex geometries (i.e., lots of vertices) or relatively simple?

Thank you, and I'll let you know if we can recreate this and get some ideas on the issue / how to address it.

-Sarah.

[NoahSlocum]

Hi Claire, 
We are still investigating the behavior you're seeing, but as a workaround for now could you try the append mode with the truncate option set to True? This would be a good workaround if you don't have any schema changes in your layer. For example:

featureServiceDf.write.format("feature-service")\
     .option("gis", "myGIS")\
     .mode("append")\
     .option("serviceUrl", myURL)\
     .option("layerId", 0)\
     .option("truncate", True)\
     .save()

I'm hoping this will work as expected and not require running the same cell twice to update the layer.