topic Re: Best Practice for Enhancing Arcpy Tools with ArcGIS API for Python to Process Feature Services in Python Questions

Best Practice for Enhancing Arcpy Tools with ArcGIS API for Python to Process Feature Services

SeanT — Mon, 30 Oct 2023 18:45:12 GMT

I am beginning to research the best way to take a suite of existing arcgis python tools and incorporating the python api so the tools can be enhanced to also work with feature services.

The tools are currently taking input fgdbs or egdbs, reading through them, some are editing features, many compare feature attributes to other 'rules' fgdbs... The tools are pretty complex, and I am trying to figure out which way to approach the task.

Ideally the existing tools would just be enhanced with the capabilities of the python api, but I suppose it could be the other way around that new tools are started with python api and they maybe then call the arcpy portions.

Thoughts, community?

Re: Best Practice for Enhancing Arcpy Tools with ArcGIS API for Python to Process Feature Services

BlakeTerhune — Mon, 30 Oct 2023 19:49:55 GMT

I'm not quite following because you mention wanting to use the API for Python to work with feature services, but you explained only how your scripts use file geodatabases. What do you plan to do with feature services?

You could take a look at this thread about modularizing Python Toolboxes for inspiration. I do something similar with our Python scheduled tasks, they import a custom library with commonly used functions.

Python Tool Management System - Esri Community

Re: Best Practice for Enhancing Arcpy Tools with ArcGIS API for Python to Process Feature Services

SeanT — Mon, 30 Oct 2023 21:20:21 GMT

I have 50+ clients under the same umbrella, but they may all do things a bit different towards the same end with similar data and schema, some use fgdbs, some use SDE, some have migrated into feature services (apologies if my use of terminology isn't exact).

Ideally each tool that is currently written with arcpy can have the python api added to it to make it even more capable, but not sure that is possible. The existing tools may be 1000+ lines of code with some coming from separate modules.

The tools scan through the input which is usually an entire dataset of facility info organized in feature datasets. The input may also be compared to a separate input real property dataset, as well as another set of rules for specific data schema rules.

So like say a client's facility data is in a feature service, they want to run the tools against their feature service instead of a fgdb or sde, but they want all the same checks, comparisons, edits, outputs to occur that the arcpy tool is doing.

I am having trouble conceptualizing this, hence my question here.

Re: Best Practice for Enhancing Arcpy Tools with ArcGIS API for Python to Process Feature Services

BlakeTerhune — Mon, 30 Oct 2023 21:32:57 GMT

So to put it another way, you want to have one tool that can accept inputs of a gdb feature class or feature service and perform the same logic?

Re: Best Practice for Enhancing Arcpy Tools with ArcGIS API for Python to Process Feature Services

SeanT — Mon, 30 Oct 2023 21:33:39 GMT

yes! if possible.

Re: Best Practice for Enhancing Arcpy Tools with ArcGIS API for Python to Process Feature Services

BlakeTerhune — Mon, 30 Oct 2023 21:40:05 GMT

Can you clarify how the feature services are hosted/accessed? (Standalone ArcGIS Server, hosted in Portal data store, uploaded to ArcGIS Online?)

Re: Best Practice for Enhancing Arcpy Tools with ArcGIS API for Python to Process Feature Services

SeanT — Mon, 30 Oct 2023 21:43:36 GMT

I'd have to find out.

How will the different storage methods affect the code?

Re: Best Practice for Enhancing Arcpy Tools with ArcGIS API for Python to Process Feature Services

BlakeTerhune — Mon, 30 Oct 2023 22:09:52 GMT

Some operations you can do on a feature class in a geodatabase might not be possible with a feature service. For example, you can't truncate/append, you would have to delete all rows, then insert all rows (for a traditional feature service hosted in standalone ArcGIS Server) and it would take considerably longer compared to a gdb fc. Also, you can't use something like arcpy.Editor or calculate field; although you can do more with GeoAnalytics tools if you have access to that.

So, it depends on a lot of different things. I would guess that "pretty complex" scripts have more than just some arcpy cursors making a few data edits. You'd have to find matching capabilities in the ArcGIS API for Python or the ArcGIS REST API directly.

Re: Best Practice for Enhancing Arcpy Tools with ArcGIS API for Python to Process Feature Services

SeanT — Mon, 30 Oct 2023 22:36:21 GMT

Thank you for the explanation... helps me put my concerns into a better perspective.

Re: Best Practice for Enhancing Arcpy Tools with ArcGIS API for Python to Process Feature Services

ChrisStiwinter1 — Fri, 05 Jan 2024 18:00:22 GMT

Hi Blake, in your post you say that you cannot truncate/append a feature service hosted in standalone GIS Server. Is the same true if the server is federated with portal?

I have been able to truncate web layers in a hosted feature service but I am having a heck of a time appending data back into the feature service. I am missing something to "connect the dots" so to speak. Seems like this would work just fine, but I have had some mixed results when trying to do this. What is the best way to update a hosted feature service in Enterprise using Python API so that we dont overwrite all the pop up settings, etc that occurs when simply republishing the service?

Also seems if a feature service has multiple layers and layer(0) in the service is truncated, then Python API cant return the layer.item for any of the remining layers in the feature service.

Any help you can give would be greatly appreciated.

Re: Best Practice for Enhancing Arcpy Tools with ArcGIS API for Python to Process Feature Services

BlakeTerhune — Fri, 05 Jan 2024 18:54:03 GMT

What method are you using when you say you "truncate web layers in a hosted feature service"? It looks like you need to use delete_features(), which could take a while depending on how many records you're working with. This will never match simplicity and performance of a database truncate though.

I see there is also an append() method for a FeatureLayer in the ArcGIS API for Python; I didn't know about this. Although, a note on append from the REST API doc:

Feature serviceAppendcapabilities must be enabled for theappendoperation to be used by nonadministrative users. Organization administrators, or the service owner, can use theappendoperation without having theAppendcapability enabled on the feature service. TheAppendcapability needs to be added to the service capabilities only if the service owner or organization administrators need to allow nonadministrative users to append data to a feature service.

I haven't used the ArcGIS API for Python or hosted feature layers in Portal much so I'm sorry I can't guide you much more. If you have a specific question about deleting features from multiple layers in a feature service, you should create a separate question for that.

Re: Best Practice for Enhancing Arcpy Tools with ArcGIS API for Python to Process Feature Services

ChrisStiwinter1 — Fri, 05 Jan 2024 19:17:23 GMT

Thanks for reply.

After signing into portal i get the itemID of the feature service using content.get(). Then assign this to a variable. Use this variable to access the layers of the feature service, once I have the layer I want to truncate, i assign it to another variable then truncate using layervariable.manager.truncate(). This works fine. It is appending that is causing issues.

Ill just keep working on it, i know i am missing something small. I am the admin/owner of the feature service so should not have problem with append permissions, I think I need to supply more parameters to the append tool even though schema matches between layers, maybe I will need to provide field mappings..

Thanks again

Re: Best Practice for Enhancing Arcpy Tools with ArcGIS API for Python to Process Feature Services

BlakeTerhune — Fri, 05 Jan 2024 19:54:31 GMT

Cool, thanks for letting me know about the truncate operation for FeatureLayer in the REST API!

Truncate (Feature Layer)—ArcGIS REST APIs | ArcGIS Developers

Re: Best Practice for Enhancing Arcpy Tools with ArcGIS API for Python to Process Feature Services

ChrisStiwinter1 — Wed, 10 Jan 2024 15:57:16 GMT

Ok, So after working on this some more, I am now able to truncate and append into a hosted feature layer in Enterprise using data from disk. My approach is to use Python Api to access the layers in the portal and truncate them. Next, I use the append function in ArcPRO to update the hosted feature classes with a feature class from disk. A little different then using the append function in Python API that is looking for source data as an item published to portal vs feature class on disk.

Use Python API to query the portal for the itemid ~ gis.content.get(itemid) set to a variable.

Once you have item id you can access the feature layers ~ use above variable to access layers i.e. variable.layers.

Once you have listing of layers in the feature service you can access them by assigning each layer to a variable . variable.layers[0] for first layer, variable.layers[1] for second layer, etc.. This will return the url endpoint of that layer.

once you have the layer url you can use the truncate/append function to update the hosted layers in the service. variable.layers.manager.truncate(), then I append in ArcPRO using a feature class that is stored on disk not in portal, thus a little different than using API for existing portal item.

variable.manager.truncate()
arcpy.management.Append(
inputs=r'C:/CustomerNotification/CN.gdb/ALL',
target=variable.url,
schema_type="TEST",
field_mapping=None,
subtype="",
expression="",
match_fields=None,
update_geometry="UPDATE_GEOMETRY")

This will work to update hosted feature classes in Enterprise with data from local disk without having to overwrite the service. Now, some caveats:

- When a feature service is a hosted service and the data is copied to the GIS Server, all the field names are changed to lowercase. If your source for the append has the same schema but some field names are uppercase, the append will run without error but no features will be added...

- This method will not cause the objectids to increase, as is the case with deleting features then using add_edits to add new features back in.

- This method runs slower than using json and parsing the source data then use REST calls to delete and add/update features. using REST to update via json strings seems to be the fastest, objectids will increase.

Just my experience, hope this can help someone else out. the lowercase fieldname change when hosting the data was causing an issue for my script, once I realized that I was on my way to success.