topic Re: republish sde data to arcgis online hosted service in ArcGIS API for Python Questions

republish sde data to arcgis online hosted service

OfraAi — Tue, 01 Apr 2025 13:25:04 GMT

Hi everyone

I have a hosted service, containing first layer + attachments, second layer with attachments and related table to the second layer with attachments for itself.

The original data is stored in SDE.

I need to publish the whole data once a month, and overwrite the whole data, using arcgis api for python.

No change in the layers structure.

I don't want to copy the data to temp FGDB before publishing, since there a lot of picture and the volume is large..

I could not find a solution to this scenario.

(by the way, using arcpy append worked in one desktop but not in the server, and argis.append does not support attachments...)

I'll appreciate any help.

Ofra

Re: republish sde data to arcgis online hosted service

Clubdebambos — Tue, 01 Apr 2025 15:02:51 GMT

Hi @OfraAi,

First question...

Are you publishing to ArcGIS Online or Portal?

Re: republish sde data to arcgis online hosted service

OfraAi — Tue, 01 Apr 2025 20:20:45 GMT

Hi @Clubdebambos

II'm publishing to ArcGIS Online.

Re: republish sde data to arcgis online hosted service

MikeVolz — Tue, 01 Apr 2025 20:24:23 GMT

Can you provide the python append that you tried to use?

Re: republish sde data to arcgis online hosted service

Clubdebambos — Wed, 02 Apr 2025 07:02:33 GMT

Hi @OfraAi,

These are the two options I generally use.

1. No code using a referenced service. The data resides in SDE and registered with Portal through Manager Registered Data Stores. Publish the feature service to portal as a referenced service. Any edits made to the SDE feature classes are automatically applied to the service and vice versa. Add the service to AGOL using a viewer user role for AGOL users to consume (Add item in AGOL via service Portal URL). Again, any edits in Portal/SDE will automatically apply to your AGOL service.

2. Use this script to publish from ArcGIS Pro to AGOL. You add your layers to a Map and the service name will take on the name of your Map. Re-run every time you want to overwrite the hosted service in AGOL.

import arcpy import os ################################################################################ ## Esri Documentation: ## https://pro.arcgis.com/en/pro-app/latest/arcpy/mapping/arcgisproject-class.htm ## https://pro.arcgis.com/en/pro-app/latest/arcpy/mapping/map-class.htm ## https://pro.arcgis.com/en/pro-app/latest/arcpy/sharing/featuresharingdraft-class.htm ## https://pro.arcgis.com/en/pro-app/latest/tool-reference/server/stage-service.htm ## https://pro.arcgis.com/en/pro-app/latest/tool-reference/server/upload-service-definition.htm ## ## ## ArcGIS Pro Version 3.2.0 ################################################################################ ################################################################################ ## INPUT REQUIRED ############################################################# ## the path to the APRX that contains the Map with the layers to publish aprx_path = r"" ## the name of the Map in the APRX that contains the layers to publish map_name = "MAP NAME" ## a folder to stage the definition files staging_fldr = r"" ## some tags for our Hosted Feature Service tags = "tag1,tag2,tag3" ## a description for our Hosted Feature Service description = "This is our description" ## a summary for our Hosted Feature Service summary = "This is our summary" ################################################################################ ## ACCESS ARCGIS PRO COMPONENTS ############################################### ## access the APRX aprx = arcpy.mp.ArcGISProject(aprx_path) ## access the correct map m = aprx.listMaps(map_name)[0] ################################################################################ ## STAGING ##################################################################### ## the service definition filenames and output paths sddraft_output_filepath = os.path.join(staging_fldr, f"{map_name}.sddraft") sd_output_filepath = os.path.join(staging_fldr, f"{map_name}.sd") ## if either of the output files already exists, then delete if arcpy.Exists(sddraft_output_filepath): arcpy.Delete_management(sddraft_output_filepath) if arcpy.Exists(sd_output_filepath): arcpy.Delete_management(sd_output_filepath) ################################################################################ ## MANIPULATE THE SD DRAFT ###################################################### ## use the Map object getWebLayerSharingDraft method ## returns a FeatureSharingDraft object sd_draft = m.getWebLayerSharingDraft( server_type = "HOSTING_SERVER", service_type = "FEATURE", service_name = map_name ) ## alter some properties of the sd_draft sd_draft.description = description sd_draft.summary = summary sd_draft.tags = tags sd_draft.allowExporting = True sd_draft.overwriteExistingService = True #sd_draft.portalFolder = "INSERT FOLDER NAME" ## create the Service Definition Draft file sd_draft.exportToSDDraft(sddraft_output_filepath) ################################################################################ ## STAGE THE SERVICE ########################################################### ## stage the service arcpy.server.StageService( in_service_definition_draft = sddraft_output_filepath, out_service_definition = sd_output_filepath ) ################################################################################ ## PUBLISH TO ARGIS ONLINE ##################################################### ## publish to agol arcpy.server.UploadServiceDefinition( in_sd_file = sd_output_filepath, in_server = "HOSTING_SERVER" ) ################################################################################ print("\nSCRIPT COMPLETE")

Original blog post here.

Edit: For the ArcPy approach, make sure you are signed into AGOL in Pro or use the below.

arcpy.SignInToPortal(portal_url,username,password)

Please le us know if either of these suit.

All the best,

Glen

Re: republish sde data to arcgis online hosted service

OfraAi — Wed, 02 Apr 2025 07:01:38 GMT

Hi @MikeVolz ,

By this article, and a case I've opened, and an existing enhancement (ENH-000131471, ENH-000173902), python api for arcgis append does not honor attachments.

Regarding using arcpy, I had problem reaching arcgis Online from different computers.

I have this script to check it, and any arcpy command can by used.

--------

import arcpy

from arcgis.gis import GIS

print('connecto to Online')

OnlineURL = "https://atikot.maps.arcgis.com"

username = 'username '

password = 'password'

gis = GIS(OnlineURL, username, password)

print('Connected User: ' + gis.users.me.username)

item_id = '1111111111'

O_Feature_Service = gis.content.get(item_id)

O_layer = O_Feature_Service.layers[0]

O_layer_url = O_layer.url

print(O_layer_url)

Olyr_query_result = O_layer.query(out_fields = ["GlobalID"])

print('arcgis: ' + str(len(Olyr_query_result.features)))

s = arcpy.SignInToPortal(OnlineURL,username,password)

layer =arcpy.management.MakeFeatureLayer(O_layer_url + "?token="+ s["token"], "aaa").getOutput(0)

print('arcpy: ' + str(arcpy.GetCount_management(layer)[0]))

print('end')

--------

Good result returns the number of features using python api and arcpy.

Bad result returns the number of features using python api but when using any arcpy command I get an erro: Parameters are not valid. error 732 Dataset https:.... does not exist or is not supported. Failed to execute.

If you have any idea why it works only on several computers and gives error on others - it would be great.

Both computers has python version 3.11.8

Ofra

Re: republish sde data to arcgis online hosted service

OfraAi — Wed, 02 Apr 2025 07:21:44 GMT

Hi @Clubdebambos Glen,

I'll try this script right now, although it uses arcpy to access arcgis online, and I'm not sure it will work. See my other comment to MikeVolz

Thank you, I'll update soon

Ofra

Re: republish sde data to arcgis online hosted service

Clubdebambos — Wed, 02 Apr 2025 09:45:58 GMT

Hi @OfraAi,

Have you tried without the "token"? If they have ArcGIS Pro installed and access to ArcPy, they more than likely don't need a token.

You don't need to MakeFeatureLayer to get the feature count, just access the URL directly.

import arcpy fl = "https://services-eu1.arcgis.com/*****/arcgis/rest/services/County Boundaries/FeatureServer/0" ## Or # fl = O_layer.url feature_count = arcpy.management.GetCount(fl) print(feature_count) >>> 26

Re: republish sde data to arcgis online hosted service

OfraAi — Wed, 02 Apr 2025 14:37:12 GMT

@Clubdebambos

The attached script just demonstrates the problem I had using arcpy with ArcGIS Online.

I've found the difference between the different environment, and it seems to be the definition of temporary storage.

when I've added:

arcpy.CreateScratchName(workspace=arcpy.env.scratchGDB)

it works.

Now I have to implement this in the whole script.

I'll update.

Thanks anyway

Ofra