topic Re: Publishing Geoprocessing Service that use Feature Service URLs in Python Questions

Publishing Geoprocessing Service that use Feature Service URLs

ZacharyHart — Mon, 11 Apr 2022 16:52:27 GMT

I've developed a script tool which I'm looking to publish as a GP service. The script leverages several feature service URLs from our Enterprise system hard-coded inputs for the script; using direct DB connections to the under lying database isn't a viable option.

Obviously, you cannot/do not register server URLs with your Data Store, but since the validation process cannot find these resources in the Data Store, it warns that the data source is not registered and will therefore be copied.

inFeatures = "https://myserver.mydomain.com/server/rest/services/Folder/FeatureServiceName/FeatureServer/1"

I thought that maybe I could simply parameterize these URLs in the Script Tool but I'm not sure that you can make these hidden or locked.

Is there anyway to go about publishing this without using a direct DB connection?

EDIT: I'm suddenly remembering something about registering a folder that contains a valid AGS connection file...hmmm

UPDATE: No dice. I saved a server connection file (.ags) to a folder registered with the data store, but I've am not able to use this input in any code. For example, making a feature layer from the REST endpoint for the layer within the feature service using the .ags file:

testLayer = arcpy.management.MakeFeatureLayer("\\\\sharedLocation\\Folder\\AGSConnections\\connectionFile.ags\\serverFolder\\featureService.FeatureServer\\layer","test")

[Note this is the format used if you drag the layer from ags connection file in the catalog pane to the Python window in Pro.]

Re: Publishing Geoprocessing Service that use Feature Service URLs

DonMorrison1 — Thu, 07 Apr 2022 15:58:45 GMT

Can you please post your code snippet to show how you are using defining the URL and using it to access the data store?

Re: Publishing Geoprocessing Service that use Feature Service URLs

ZacharyHart — Thu, 07 Apr 2022 16:19:53 GMT

I'm simply referencing them to the Feature Service REST endpoints like so:

inFeatures = "https://myserver.mydomain.com/server/rest/services/Folder/FeatureServiceName/FeatureServer/1"

Re: Publishing Geoprocessing Service that use Feature Service URLs

DonMorrison1 — Fri, 08 Apr 2022 00:02:56 GMT

If it is the consolidation process that is changing the string to something incorrect (I've seen this happen with directory paths but not urls), then I've been able to get around problems like that by breaking up the string literal. I'm not sure if this is relevant to your situation......

Here is an example of how I did it

# To get around a problem with consolidation we break the template directory name apart # to trick the consolidator so it doesn't try to change the path (for shared projects the # consolidator puts the files in the right place but generates the wrong path literals) template_rptx = os.path.dirname(__file__) + os.path.sep + 'report' + '_' + 'templates' + os.path.sep + 'CCAA_Report.rptx'

Re: Publishing Geoprocessing Service that use Feature Service URLs

ZacharyHart — Fri, 08 Apr 2022 00:18:05 GMT

It's not during consolidation, it's during validation because it's evaluating the URL as a data source registered with the data store or not. And since the data store does not register valid REST endpoints but only direct DB connections, this will raise an error and advise to copy the data to the server (a silly circular reference).

That said, the faking the string as you suggested is intriguing and I will try that to see if it sneaks by the validation.

It's odd that this isn't accounted for since Python web API would happily use a feature service URL even in a federated environment and you could share (and possibly publish, I've not tried) that notebook but for a script tool you cannot.

Obviously the script tool works just fine while using Pro (it's not necessary to add the layers to your session but only that Pro is logged into Portal).

I can flip this to direct GDB connections but it will be so so slow (remote DB in the cloud) for the initial run of the script tool just so that I can capture the result and use that to publish the GP Service.

Re: Publishing Geoprocessing Service that use Feature Service URLs

ZacharyHart — Wed, 01 Jun 2022 17:01:12 GMT

I'm unfortunately still not having any luck with this.

Re: Publishing Geoprocessing Service that use Feature Service URLs

ZacharyHart — Fri, 01 Jul 2022 13:19:55 GMT

Is there anyone from Esri willing to answer this? We have a number of GP tools we'd like to roll out to our users that are currently working in Pro using FS within the code but cannot be successfully published.

Why wouldn't you want to leverage the speed of FS in your code vs direct GDB connections?

Re: Publishing Geoprocessing Service that use Feature Service URLs

ZacharyHart — Thu, 17 Aug 2023 15:29:36 GMT

Does anyone have any suggestions regarding this?

Does anyone utilize Feature Services as an input to GP Tools??

Here's what Chat GPT 4 had to offer:

It sounds like you're facing an issue while trying to publish a geoprocessing tool to ArcGIS Server that uses a feature service as input. The problem arises because ArcGIS Server doesn't support registering a feature service with ArcGIS Datastore, which leads to a failure during the validation process. Here are a few steps you can consider to resolve this issue:

1. **Use a Geoprocessing Service**: Instead of trying to publish the geoprocessing tool with the feature service as input directly, you could create a separate geoprocessing service that takes the feature service as input and then use that geoprocessing service as a tool within the main tool you're trying to publish.

2. **Data Extract**: Depending on the nature of your geoprocessing task, you could use the feature service as a reference but extract the necessary data to a file geodatabase or other supported data source that can be used in the geoprocessing tool. This extracted data could then be included with your geoprocessing tool when you publish it.

3. **Temporary Feature Class**: If possible, you could create a temporary feature class within a supported geodatabase (like a file geodatabase) and populate it with the features from the feature service. This temporary feature class could be used as input for your geoprocessing tool.

4. **Data Replication**: If your workflow requires frequent updates to the feature service, you could consider setting up a data replication process where the feature service's data is periodically copied to a registered data source that is compatible with ArcGIS Server.

5. **Contact Esri Support**: If you're still facing issues, it might be a good idea to reach out to Esri's technical support. They can provide you with more specific guidance and troubleshoot your scenario to find the best solution.

Remember that the specifics of your workflow and environment can influence the best approach to take. Choose the option that aligns most closely with your requirements and available resources.

Re: Publishing Geoprocessing Service that use Feature Service URLs

cepsgis — Mon, 25 Mar 2024 17:51:45 GMT

@ZacharyHart , I know this is a late but I put my urls in a text file and read them from it. This excludes it from the tool validation.

Re: Publishing Geoprocessing Service that use Feature Service URLs

Ed-Conrad — Wed, 19 Feb 2025 21:59:26 GMT

Edit - 2/19/25

I've completely updated my answer. Prior answer I put them as parameter objects in the Python toolbox - this worked initially, but subsequently, I was getting new errors
.

Simpler approach:

I publish the Python toolbox tool from ArcGIS Pro and get a successful run in my ArcGIS Pro History. Then prior to publishing the tool as a web tool, I comment out all the code in execute() where the URLs are located - minus a pass statement. Then I publish the tool to Portal. I subsequently edit the .pyt file directly where it was copied to ArcGIS Server and uncomment out the code.

C:\arcgisserver\directories\arcgissystem\arcgisinput\[folder]\[web tool/GP tool name].GPServer\extracted\p20\report\[my python toolbox file].pyt

Re: Publishing Geoprocessing Service that use Feature Service URLs

DougBrowning — Mon, 07 Oct 2024 21:58:27 GMT

Did you ever figure this out @ZacharyHart ?

Re: Publishing Geoprocessing Service that use Feature Service URLs

samc-gis — Fri, 14 Feb 2025 17:42:23 GMT

I experienced this issue myself when trying to set up a local script as a GP service - I foolishly thought that the validation would "just know" that the hard-coded feature services were feature services and not local data and therefore would not attempt to copy the data when publishing. I attempted to publish the script with the hard-coded URLs and I ended up having to kill the publishing job as we all know it attempted to copy the data to the server despite no actual local data being referenced in the script.

The issue with validation flagging feature service URLs as file directories can be avoided by storing all URLs in a YAML configuration file. Instead of embedding/hard-coding the URLs directly in the script or tool, reference them dynamically by reading from the YAML file at runtime. This way, the only file that gets copied with the GP service is the small YAML file, avoiding any validation errors related to directory structures and drastically reducing the time to publish the GP service.

For example, you can structure the YAML file like this:

portal: master_fs: https://yourserver.com/FeatureServer/0 feature_services: other_fs1: "https://yourserver.com/FeatureServer/1" other_fs2: "https://yourserver.com/FeatureServer/2" other_fs3: "https://yourserver.com/FeatureServer/3"

Then, in your script, load the URLs dynamically:

import yaml def read_config(config_path): with open(config_path, 'r') as file: return yaml.safe_load(file) config = read_config("config.yaml") master_feature_service = config["portal"]["master_fs"] feature_service_urls = config["feature_services"]

This approach ensures that no hardcoded paths cause validation issues and makes it easier to update URLs without modifying the script itself. You can modify the YAML file published with the GP service by going to the file directory for the GP service on your server if you need to add/remove variables.

Hope this helps anyone that has been struggling with this issue!

Re: Publishing Geoprocessing Service that use Feature Service URLs

DougBrowning — Fri, 14 Feb 2025 18:26:18 GMT

Yes or regular old text file works also. In the end now I just comment out my script totally, publish, then on the server uncomment it back. Works way faster and then nothing gets repathed at all. Took some convincing with IT but now I can skip all validation. Script is much cleaner now and I can publish in like a min vs the 20 minutes it was taking to repath on me. Then going forward I just make changes to the script on the server vs republishing. I know its not great to change production like that but I only really do it for dev scripts. Light years faster to troubleshoot the 500 other things that break in GP services.