What is the smoothest way to go from data in memory to publishing on ArcGIS Enterprise?

Question

Sorry for the basic question, but it seems like just about all of the publication examples on the ArcGIS API for Python docs presume that content is going to be published from a flat file sitting on a drive somewhere (e.g., here's a literal some-name.geojson file with 4 points from my camping trip that I want to publish to AGOL).

The dynamic publishing workflows documented (e.g., spatial data frame to hosted feature layer) presume that Notebook Server is part of the equation.  So, when I run a script meant to simply push a spatial data frame of a few points into ArcGIS Enterprise from VS Code or command line, it just throws up and says "Exception: Job failed." (I guess because it's not connecting with any authenticated session).

So my question is: how do I go from dynamic, non-file based spatial content in memory (GeoJSON or spatial data frame) to a newly published hosted feature class on ArcGIS Enterprise, without using Notebook Server and without writing out a bunch of intermediate files?

EricEagle · Answer

@EarlMedina Here's the traceback.  This is being run on an account with Administrator permissions.  I've grabbed the log output as well and put it below:

Traceback (most recent call last):
  File "x:\wfs2age\scratch2.py", line 30, in <module>
    pub_data.spatial.to_featurelayer('publish_test', folder='testings')
  File "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\features\geo\_accessor.py", line 2912, in to_featurelayer
    result = content.import_data(
             ^^^^^^^^^^^^^^^^^^^^
  File "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\gis\__init__.py", line 8591, in import_data
    return _cm_helper.import_as_item(self._gis, df, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\gis\_impl\_content_manager\_import_data.py", line 247, in import_as_item
    file_item, new_item = _create_file_item(gis, df, file_type, **kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\gis\_impl\_content_manager\_import_data.py", line 147, in _create_file_item
    new_item = file_item.publish(
               ^^^^^^^^^^^^^^^^^^
  File "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\gis\__init__.py", line 16279, in publish
    return job.result()
           ^^^^^^^^^^^^
  File "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\concurrent\futures\_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
  File "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\concurrent\futures	hread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\gis\__init__.py", line 16732, in _publish
    serviceitem_id = self._check_publish_status(ret, folder)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\gis\__init__.py", line 17092, in _check_publish_status
    raise Exception("Job failed.")
Exception: Job failed.

Here's the log from the portal admin side (it's Enterprise 10.9.1):

Failed to publish item 'https://arcgisenterprise.local/arcgis1/rest/services/Hosted/a2a63a/FeatureServer ' (id:03497fbee93f44128082720e68cdc031). ERROR 999999: Something unexpected caused the tool to fail. Contact Esri Technical Support (http://esriurl.com/support) to Report a Bug, and refer to the error help for potential solutions or workarounds. This version of the Geodatabase client is incompatible with the dataset and cannot open it. [a7d6ffa8] This version of the Geodatabase client is incompatible with the dataset and cannot open it. [dataset name: a7d6ffa8] Operation on file failed. Unknown error. Failed to execute (Publish Portal Service). Failed.. Job ID: 'j73c70356881c4a9fb21c7357f58e7eae'

EarlMedina · Answer

There's not a stacktrace pointing to a problem? It's hard to say what's going on, but I assure you that while that method is not perfect it does work. Perhaps a different user account with publishing rights or different data may yield different results.

EricEagle · Answer

@EarlMedina So, here's what I'm doing, with the default ArcGIS Pro python environment:

from arcgis.features import FeatureSet, GeoAccessor
from arcgis.gis import GIS
import arcpy
import os
import pandas as pd

# Connect
conn = {
    "username": "myusername",
    "password": "mypassword",
    "portal_url": "https://my-enterprise-instance/portal"}

gis = GIS(conn["portal_url"], conn["username"], conn["password"], verify_cert=False)  # This works

# Data
pub_fgdb = 'Y:/ArcGIS/Projects/Upload_Test/Upload_Test.gdb'
pub_lyr = os.path.join(pub_fgdb, 'topublish')

pub_data = pd.DataFrame.spatial.from_featureclass(pub_lyr) # works
pub_data.spatial.to_featurelayer('publish_test', folder='testings')  # Yields "Job failed."

@