Select to view content in your preferred language

Publishing Geoprocessing Service that use Feature Service URLs

1283
9
04-07-2022 08:28 AM
ZacharyHart
Honored Contributor

I've developed a script tool which I'm looking to publish as a GP service. The script leverages several feature service URLs from our Enterprise system hard-coded inputs for the script; using direct DB connections to the under lying database isn't a viable option.

Obviously, you cannot/do not register server URLs with your Data Store, but since the validation process cannot find these resources in the Data Store, it warns that the data source is not registered and will therefore be copied.

 

inFeatures = "https://myserver.mydomain.com/server/rest/services/Folder/FeatureServiceName/FeatureServer/1"

 

I thought that maybe I could simply parameterize these URLs in the Script Tool but I'm not sure that you can make these hidden or locked.

Is there anyway to go about publishing this without using a direct DB connection?

EDIT: I'm suddenly remembering something about registering a folder that contains a valid AGS connection file...hmmm

UPDATE: No dice. I saved a server connection file (.ags) to a folder registered with the data store, but I've am not able to use this input in any code. For example, making a feature layer from the REST endpoint for the layer within the feature service using the .ags file:

 

testLayer = arcpy.management.MakeFeatureLayer("\\\\sharedLocation\\Folder\\AGSConnections\\connectionFile.ags\\serverFolder\\featureService.FeatureServer\\layer","test")

 

[Note this is the format used if you drag the layer from ags connection file in the catalog pane to the Python window in Pro.]

0 Kudos
9 Replies
DonMorrison1
Frequent Contributor

Can you please post your code snippet to show how you are using defining the URL and using it to access the data store?

0 Kudos
ZacharyHart
Honored Contributor

I'm simply referencing them to the Feature Service REST endpoints like so:

inFeatures = "https://myserver.mydomain.com/server/rest/services/Folder/FeatureServiceName/FeatureServer/1"
0 Kudos
DonMorrison1
Frequent Contributor

If it is the consolidation process that is changing the string to something incorrect (I've seen this happen with directory paths but not urls), then I've been able to get around problems like that by breaking up the string literal. I'm not sure if this is relevant to your situation......

Here is an example of how I did it

    # To get around a problem with consolidation we break the template directory name apart
    # to trick the consolidator so it doesn't try to change the path (for shared projects the
    # consolidator puts the files in the right place but generates the wrong path literals)
    template_rptx = os.path.dirname(__file__) + os.path.sep +  'report' + '_' + 'templates' + os.path.sep +  'CCAA_Report.rptx'

 

0 Kudos
ZacharyHart
Honored Contributor

It's not during consolidation, it's during validation because it's evaluating the URL as a data source registered with the data store or not. And since the data store does not register valid REST endpoints but only direct DB connections, this will raise an error and advise to copy the data to the server (a silly circular reference).

That said, the faking the string as you suggested is intriguing and I will try that to see if it sneaks by the validation.

It's odd that this isn't accounted for since Python web API would happily use a feature service URL even in a federated environment and you could share (and possibly publish, I've not tried) that notebook but for a script tool you cannot. 

Obviously the script tool works just fine while using Pro (it's not necessary to add the layers to your session but only that Pro is logged into Portal).

I can flip this to direct GDB connections but it will be so so slow (remote DB in the cloud) for the initial run of the script tool just  so that I can capture the result and use that to publish the GP Service. 

0 Kudos
ZacharyHart
Honored Contributor

I'm unfortunately still not having any luck with this.

0 Kudos
ZacharyHart
Honored Contributor

Is there anyone from Esri willing to answer this? We have a number of GP tools we'd like to roll out to our users that are currently working in Pro using FS within the code but cannot be successfully published.

Why wouldn't you want to leverage the speed of FS in your code vs direct GDB connections?

0 Kudos
ZacharyHart
Honored Contributor

Does anyone have any suggestions regarding this? 

Does anyone utilize Feature Services as an input to GP Tools??

Here's what Chat GPT 4 had to offer:

It sounds like you're facing an issue while trying to publish a geoprocessing tool to ArcGIS Server that uses a feature service as input. The problem arises because ArcGIS Server doesn't support registering a feature service with ArcGIS Datastore, which leads to a failure during the validation process. Here are a few steps you can consider to resolve this issue:

1. **Use a Geoprocessing Service**: Instead of trying to publish the geoprocessing tool with the feature service as input directly, you could create a separate geoprocessing service that takes the feature service as input and then use that geoprocessing service as a tool within the main tool you're trying to publish.

2. **Data Extract**: Depending on the nature of your geoprocessing task, you could use the feature service as a reference but extract the necessary data to a file geodatabase or other supported data source that can be used in the geoprocessing tool. This extracted data could then be included with your geoprocessing tool when you publish it.

3. **Temporary Feature Class**: If possible, you could create a temporary feature class within a supported geodatabase (like a file geodatabase) and populate it with the features from the feature service. This temporary feature class could be used as input for your geoprocessing tool.

4. **Data Replication**: If your workflow requires frequent updates to the feature service, you could consider setting up a data replication process where the feature service's data is periodically copied to a registered data source that is compatible with ArcGIS Server.

5. **Contact Esri Support**: If you're still facing issues, it might be a good idea to reach out to Esri's technical support. They can provide you with more specific guidance and troubleshoot your scenario to find the best solution.

Remember that the specifics of your workflow and environment can influence the best approach to take. Choose the option that aligns most closely with your requirements and available resources.

0 Kudos
cepsgis
Occasional Contributor

@ZacharyHart , I know this is a late but I put my urls in a text file and read them from it. This excludes it from the tool validation.  

0 Kudos
Ed-Conrad
Occasional Contributor

I stumbled on your post trying to find a solution myself! 

I ended up hardcoding the web service as a Parameter.  This gets it past the publish a GP Service validation where ArcGIS Pro detects the URL as a data source not registered with the data store (this was driving me crazy too!).  In my case, a Map Service was sufficient for my needs to query information for generating an Excel report.

Parameter objects have an enabled property.  When I set it to False, it removes the parameter from being shown in ArcGIS Pro and when you go to publish the Web Tool/GP service, you are no longer harassed by that validation check inspecting all the strings, including URLs.  However, how you might hide it from your users in a web map/app would depend on what it is (enabled doesn't really matter outside of it being used in ArcGIS Pro).

For instance, going straight to the ArcGIS Server REST endpoint for the GP Service, I still see all the parameters (.enabled = False has zero impact here).  In my case, we're doing this for a client that has VertiGIS, so I create a DisplayForm that only has the parameters I want to show, but looking at ArcGIS Web App Builder and it's Geoprocessing Service, it has a Visible property that could be quickly unchecked for the ones that should remain hidden (screen capture).

def getParameterInfo(self):
    """Define the tool parameters."""
    start_date = arcpy.Parameter(
        displayName='Start Date',
        name='start_date',
        datatype='GPDate',
        parameterType='Required',
        direction='Input')
    current_year = dt.datetime.now().year
    start_date.value = dt.datetime(current_year, 1, 1)

    end_date = arcpy.Parameter(
        displayName='End Date',
        name='end_date',
        datatype='GPDate',
        parameterType='Required',
        direction='Input')
    end_date.value = dt.datetime.now()

    # Map Service made into parameter to get around Web Tool publish validation check where ArcGIS inspects all
    # strings including URLs and checks if they're data sources.
    # If they are, it warns that the service will be copied to the server since it is not a registered data store...
    map_service = arcpy.Parameter(
        displayName='Map Service',
        name='map_service',
        datatype='GPString',
        parameterType='Required',
        direction='Input')
    map_service.value = 'https://[domain]/arcgis/rest/services/[folder]/[service name]/MapServer'
    map_service.enabled = False  # prevents it from being seen

    assignments_layer_id = arcpy.Parameter(
        displayName='VertiGIS Assignments Table',
        name='assignments_layer_id',
        datatype='GPLong',
        parameterType='Required',
        direction='Input')
    assignments_layer_id.value = 4
    assignments_layer_id.enabled = False

    deployments_layer_id = arcpy.Parameter(
        displayName='VertiGIS Deployments Table',
        name='deployments_layer_id',
        datatype='GPLong',
        parameterType='Required',
        direction='Input')
    deployments_layer_id.value = 5
    deployments_layer_id.enabled = False

    # This will be passed from VertiGIS Workflow by a web request, generating a token at time report is requested.
    token = arcpy.Parameter(
        displayName='Token',
        name='token',
        datatype='GPString',
        parameterType='Optional',  # if not provided, simply runs and doesn't do much!
        direction='Input')

    report = arcpy.Parameter(
        displayName='Excel Report',
        name='excel_report',
        datatype='DEFile',
        parameterType='Derived',
        direction='Output')

    return [start_date, end_date, map_service, assignments_layer_id, deployments_layer_id, token, report]


def execute(self, parameters, messages):
    map_service = [param for param in parameters if param.name == 'map_service'][0].value
    assignments_layer_id = [param for param in parameters if param.name == 'assignments_layer_id'][0].value
    deployments_layer_id = [param for param in parameters if param.name == 'deployments_layer_id'][0].value

    [...]

 

0 Kudos