topic Re: Field not get calculated in geoprocessing service in Python Questions

Field not get calculated in geoprocessing service

ionarawilson1 — Tue, 01 Oct 2013 15:57:53 GMT

This script works great in ArcMap, updating the field EditField. However when I create a geoprocessing service from the result and run it, the field does not get update, although I get no error messages when running it. Does anybody would have any idea why the field would not get updated with the geoprocessing service? Thank you

# -*- coding: utf-8 -*- # --------------------------------------------------------------------------- # calcfeatures_script.py # Created on: 2013-09-30 15:22:46.00000 #   (generated by ArcGIS/ModelBuilder) # Usage: calcfeatures_script <Selecting_Features> <value>  # Description:  # test # ---------------------------------------------------------------------------  # Import arcpy module import os, sys, arcpy, traceback, arcgisscripting  # Script arguments Selecting_Features = arcpy.GetParameterAsText(0) value = arcpy.GetParameterAsText(1) Stewardship = arcpy.GetParameterAsText(2) arcpy.AddMessage("TEST") arcpy.AddMessage(value)   # Local variables: Input_Points = Stewardship #Final_Output = value  Input_Points_Layer = "SamplePoints_Layer"  # Process: Make Feature Layer arcpy.MakeFeatureLayer_management(Input_Points, Input_Points_Layer) # Process: Select Layer By Location arcpy.SelectLayerByLocation_management(Input_Points_Layer, "INTERSECT", Selecting_Features, "", "NEW_SELECTION")  # Process: Calculate Field    with arcpy.da.UpdateCursor(Input_Points_Layer, ("City")) as rows:         # row comes back as a tuple in the order specified here, so Office is row[0], Forester is row[1]         for row in rows:             row[0] = value                         rows.updateRow(row)

Re: Field not get calculated in geoprocessing service

T__WayneWhitley — Tue, 01 Oct 2013 16:45:13 GMT

...just a shot at this, have you verified your data is accessible by Server?

Making your data accessible to ArcGIS Server
Services » ArcGIS for Server (Windows) » Publishing services » How to publish services
http://resources.arcgis.com/en/help/main/10.1/index.html#//01540000039r000000

...and have you set up the script params via the Service Editor?

Overview of geoprocessing task settings
Services » ArcGIS for Server (Windows) » Publishing services » Types of services » Geoprocessing services » Publishing geoprocessing services
http://resources.arcgis.com/en/help/main/10.1/index.html#//01540000058s000000

Re: Field not get calculated in geoprocessing service

ionarawilson1 — Sun, 12 Dec 2021 04:46:10 GMT

Thank you Wayne. I checked everything you mentioned and everything seems fine. What I ended up doing is putting the file in a sde geodatabase and instead of asking the user to choose the file, I created a constant path to the SDE geodatabase. It worked like a charm. I am not sure why the geodatabase that is not in SDE is not being updated. I will try again with a constant path and see if that works. But I am happy it is working now, because my final goal was to edit a file in a SDE geodatabase (I thought it would be difficult so I started testing with a local file geodatabase). Thank you!

# -*- coding: utf-8 -*-
# ---------------------------------------------------------------------------
# calcfeatures_script.py
# Created on: 2013-09-30 15:22:46.00000
#   (generated by ArcGIS/ModelBuilder)
# Usage: calcfeatures_script <Selecting_Features> <value> 
# Description: 
# test
# ---------------------------------------------------------------------------

# Import arcpy module
import os, sys, arcpy, traceback, arcgisscripting
arcpy.env.workspace = "Database Connections\\sars.sde\\"
# Script arguments
Selecting_Features = arcpy.GetParameterAsText(0)
value = arcpy.GetParameterAsText(1)




# Local variables:
Input_Points = "sars.dbo.CITYTEST"
#Final_Output = value

Input_Points_Layer = "SamplePoints_Layer"

# Process: Make Feature Layer
arcpy.MakeFeatureLayer_management(Input_Points, Input_Points_Layer)
# Process: Select Layer By Location
arcpy.SelectLayerByLocation_management(Input_Points_Layer, "INTERSECT", Selecting_Features, "", "NEW_SELECTION")

# Process: Calculate Field



with arcpy.da.UpdateCursor(Input_Points_Layer, ("City")) as rows:
        # row comes back as a tuple in the order specified here, so Office is row[0], Forester is row[1]
        for row in rows:
            row[0] = value
           
            rows.updateRow(row)

Re: Field not get calculated in geoprocessing service

ionarawilson1 — Sun, 12 Dec 2021 04:46:12 GMT

I tried doing the same with the file geodatabase and it worked. For some reason it need to have a constant path and a workspace defined. Now it is updating perfectly 🙂

# -*- coding: utf-8 -*-
# ---------------------------------------------------------------------------
# calcfeatures_script.py
# Created on: 2013-09-30 15:22:46.00000
#   (generated by ArcGIS/ModelBuilder)
# Usage: calcfeatures_script <Selecting_Features> <value> 
# Description: 
# test
# ---------------------------------------------------------------------------

# Import arcpy module
import os, sys, arcpy, traceback, arcgisscripting
#arcpy.env.workspace = "Database Connections\\sars.sde\\"
arcpy.env.workspace = "d:/ArcGISData/SARS/Python_10April2013/SARS.gdb"
# Script arguments
Selecting_Features = arcpy.GetParameterAsText(0)
value = arcpy.GetParameterAsText(1)




# Local variables:
#Input_Points = "sars.dbo.CITYTEST"
Input_Points = "Stewardship"
#Final_Output = value

Input_Points_Layer = "SamplePoints_Layer"

# Process: Make Feature Layer
arcpy.MakeFeatureLayer_management(Input_Points, Input_Points_Layer)
# Process: Select Layer By Location
arcpy.SelectLayerByLocation_management(Input_Points_Layer, "INTERSECT", Selecting_Features, "", "NEW_SELECTION")

# Process: Calculate Field



with arcpy.da.UpdateCursor(Input_Points_Layer, ("City")) as rows:
        # row comes back as a tuple in the order specified here, so Office is row[0], Forester is row[1]
        for row in rows:
            row[0] = value
           
            rows.updateRow(row)

Re: Field not get calculated in geoprocessing service

T__WayneWhitley — Tue, 01 Oct 2013 18:46:47 GMT

I am experimenting with this as well....I thought this was an important find:

http://forums.arcgis.com/threads/77380-Use-feature-class-with-a-GP-service#4

...and I quote the response below from Kevin Hibma (thank you Kevin) and I thought that clever as Kevin puts it, to 'mash up paths' or to run the model/script against a layer (rather than the hard-coded path). Finally, the database must be registered as datastore.

"...the user of the service is basically trying to remotely administer a geodatabase and the featureclasses inside: you'll have to create a sort of look up and pass just simple names to the service. In this scenario I assume you'll be working against 1 to any number of featureclasses. Here you have 2 ways to do this:
1) Accept a NAME from the user, the service then uses the name to mash up paths, find the featureclass in a geodatabase, and perform the operation. (You can do this in a model, but a script might be easier)
2) Drag all the layers that a user may want to act upon into your MXD. Run your model against a layer. Publish. Inside the Service Editor, change the parameter to choice list. Finish publishing. When the user consumes the service they pick 1 or more layers from the parameter control.

Both of these methods would require registering the database as a datastore..."

Enjoy,
Wayne

Re: Field not get calculated in geoprocessing service

ionarawilson1 — Sun, 12 Dec 2021 04:46:15 GMT

I need to read again and again to understand this part about mashin up paths. Do you have any examples you can show me?

Can you tell me why my example does not work with the geoprocessing service (it works in ArcMap) although, the other models worked when the input was not a parameter? Thank you!!

# Import arcpy module
import os, sys, arcpy, traceback, arcgisscripting
arcpy.env.workspace = "Database Connections\\sars.sde\\"
#arcpy.env.workspace = "d:/ArcGISData/SARS/Python_10April2013/SARS.gdb"

# Script arguments
Selecting_Features = arcpy.GetParameterAsText(0)
value = arcpy.GetParameterAsText(1)
stewardship =  arcpy.GetParameterAsText(2)

# Local variables:
#Input_Points = "sars.dbo.CITYTEST"
Input_Points = stewardship
#Final_Output = value

Input_Points_Layer = "SamplePoints_Layer"

# Process: Make Feature Layer
arcpy.MakeFeatureLayer_management(Input_Points, Input_Points_Layer)
# Process: Select Layer By Location
arcpy.SelectLayerByLocation_management(Input_Points_Layer, "INTERSECT", Selecting_Features, "", "NEW_SELECTION")

# Process: Calculate Field



# Process: Calculate Field
#arcpy.CalculateField_management(Input_Points_Layer, "City, '"' + value + '"', "PYTHON_9.3")

with arcpy.da.UpdateCursor(Input_Points_Layer, ("City")) as rows:
        # row comes back as a tuple in the order specified here, so Office is row[0], Forester is row[1]
        for row in rows:
            row[0] = value
           
            rows.updateRow(row)

Re: Field not get calculated in geoprocessing service

T__WayneWhitley — Tue, 01 Oct 2013 20:41:30 GMT

Certainly, at the post I gave you that tereshenkov made (after Kevin's answer) that includes his 'mash-up', it looks like the below where in fact he used a kind of variable substitution to feed to the Truncate tool (the tool is irrelevant, I think you are simply passing in a text value).

r'C:\sde_at_DB_101.sde\%s' %GetPolyFC

So his text value passed in called 'GetPolyFC' (actually it is called something else initially, but he passed the val to a new variable, so it's the same thing).

The tool he is executing I suppose needs the full pathname i.e. including the workspace, which in his case is the server accessible pathname to his sde file containing the db params - that's actually the 1st part of the string above:

r'C:\sde_at_DB_101.sde'

So the so-called mash-up is a Python trick to substitute %s with the value of GetPolyFC, understand? The text passed is the feature class name - and if the user types that in wrong, he even checks for existence and traps that error (if you look at his code).

Hope that helps.

Enjoy,
Wayne

EDIT: So, I haven't tested this yet (I will be conducting this simple test tomorrow), but I think that would be called a 'User Defined Value', covered here:

Input modes and parameter data types
Desktop » Geoprocessing » Sharing geoprocessing workflows » Sharing workflows with geoprocessing services » Publishing geoprocessing services
http://resources.arcgis.com/en/help/main/10.1/index.html#/Input_modes_and_parameter_data_types/005700000062000000/

Re: Field not get calculated in geoprocessing service

T__WayneWhitley — Sun, 12 Dec 2021 04:46:18 GMT

So, if this helps at all, the term 'mash-up' is just in this case a way to describe combining text to make valid pathnames...not to be confused with how the term is used to combine web services to make maps, sorry 'bout that if confusing.

This simple code illustration should help maybe:

>>> GetPolyFC = 'my user-defined value'
>>> workspace = r'\\myServer\myGDB.sde'
>>> theFullPath = workspace + r'\%s' %GetPolyFC

>>> print '"' + theFullPath + '"'
"\\myServer\myGDB.sde\my user-defined value"


 
OR MAYBE THIS, IF YOU LIKE IT BETTER:
(either way, pathnames are assembled out of text passed in)

>>> import os
>>> os.path.join(workspace, GetPolyFC)

'\\\\myServer\\myGDB.sde\\my user-defined value'
>>>

Re: Field not get calculated in geoprocessing service

ionarawilson1 — Sun, 12 Dec 2021 04:46:22 GMT

Thank you for sending this and for the thorough explanation. My script is in a toolbox, so I was trying to set that parameter. I tried feature set, feature class and string. What would be the type of that user defined parameter? I first tried as a feature set and got an error when trying to run the gp (of course because it goes to the memory so arcgis server can't find it. Then I tried as a feature class. The gp runs ok, but it does not upated the field. Then lastly, I tried to set it as a string, and I got a big error when trying to run the gp. So I guess I don't know how to set up a user-defined parameter. Thank you!

Traceback (most recent call last): File "D:\arcgisserver\directories\arcgissystem\arcgisinput\CalcFeaturesStewardshipParam4.GPServer\extracted\v101\python_10april2013\calcfeatures_script._forStewardship.py", line 41, in arcpy.MakeFeatureLayer_management("Database Connections\\Connection to tfsgis-iisd01IWILSON.sde\\%s" %Input_Points , Input_Points_Layer) File "c:\program files\arcgis\server\arcpy\arcpy\management.py", line 5748, in MakeFeatureLayer raise e ExecuteError: Failed to execute. Parameters are not valid. ERROR 000732: Input Features: Dataset Database Connections\Connection to tfsgis-iisd01IWILSON.sde\sars.DBO.Stewardship does not exist or is not supported Failed to execute (MakeFeatureLayer). Failed to execute (CalcFeaturesStewardship). Failed to execute (CalcFeaturesStewardship).

import os, sys, arcpy, traceback, arcgisscripting
#arcpy.env.workspace = "Database Connections\\sars.sde\\"
arcpy.env.workspace =  "Database Connections\\Connection to tfsgis-iisd01IWILSON.sde"

# Script arguments
Selecting_Features = arcpy.GetParameterAsText(0)
value = arcpy.GetParameterAsText(1)
stewardship =  arcpy.GetParameterAsText(2)




# Local variables:
#Input_Points = "sars.dbo.CITYTEST"
Input_Points = stewardship
#Final_Output = value

Input_Points_Layer = "SamplePoints_Layer"

# Process: Make Feature Layer
arcpy.MakeFeatureLayer_management("Database Connections\\Connection to tfsgis-iisd01IWILSON.sde\\%s" %Input_Points , Input_Points_Layer)
# Process: Select Layer By Location
arcpy.SelectLayerByLocation_management(Input_Points_Layer, "INTERSECT", Selecting_Features, "", "NEW_SELECTION")

# Process: Calculate Field



# Process: Calculate Field
#arcpy.CalculateField_management(Input_Points_Layer, "City, '"' + value + '"', "PYTHON_9.3")

with arcpy.da.UpdateCursor(Input_Points_Layer, ("City")) as rows:
        # row comes back as a tuple in the order specified here, so Office is row[0], Forester is row[1]
        for row in rows:
            row[0] = value
           
            rows.updateRow(row)

Re: Field not get calculated in geoprocessing service

T__WayneWhitley — Wed, 02 Oct 2013 15:26:05 GMT

I see you're using Database Connections which likely cannot be resolved by ArcGIS Server....that is simply a connection file stored in your user directory. Notice how in the other thread referenced, he has copied (or created another one) it to a place on the C drive of the server where it is accessible. You can put it where you like, just your GIS Server has to be able to 'see' it...

That's the 1st thing I saw. If you're taking the text mashup approach, you should be passing text in the tool param, and define it in the service editor as user-defined.

Re: Field not get calculated in geoprocessing service

ionarawilson1 — Wed, 02 Oct 2013 15:48:18 GMT

Ah, I see it! It totally makes sense now that you explained. Also about arcgis server being able to see it, If I am editing a feature service, I need to create a local copy. I don't have a choice of where I can save it. It gets saved to a folder in the user directory. Is there a way for arcgis server to see that? It is because my final objective is to have a web app to edit features in a feature service. But the web app will run a geoprocessing service. However to test the gp, I cannot run the data with the feature service without first having a local copy. However the data that was copied to the local folder cannot be copied to the server (this is the message I get when I try to publish it) so I am not sure how to publish a geoprocessing service that will be editing a feature service. Is there any way this can be done? How do people run geoprocessing services in web apps when trying to edit a feature in a sde geodatabase? I can't find any information about it.

Let me start a new thread with this question so you can get more points for it. Thanks

Thank you so much for your help. Your knowledge has helped me tremendously. I have been trying to figure out this for days and there is not really a training course I could take to learn all this.

Re: Field not get calculated in geoprocessing service

T__WayneWhitley — Wed, 02 Oct 2013 18:01:21 GMT

Let me be honest, I am learning this with you... so it is as hit or miss, read, change something, and test again for me as it is for you. I try to follow along here on the forums to make sure I am more and more engaged. Certainly I contribute here or there where my reach has exceeded my knowledge, or else I have become rusty and forgetful -- either way, let's just say I'm stretching to make more room for increased knowledge, and to help each other attain it.

You've got me stumped at the moment, just read so far that you can set up your gp service to read layers I am assuming are from a published map, to then pass to the server script task, but as I understand it not quite the same as a feature editing service, but I'll have to read up more. I'll be watching out and following some of your new threads....maybe you can entice Kevin to contribute with a rich answer! I will tune in for that.

Enjoy,
Wayne

Re: Field not get calculated in geoprocessing service

ionarawilson1 — Wed, 02 Oct 2013 18:22:57 GMT

Thank you Wayne. I have started writing the web app in javascript and I am using a edit widget that can be used to enter new features in the feature service. Everything works great, I can already create features and they get automatically created in the sql database. Since the data in the feature service comes from the same sde I have been using, I am guessing I can run a geoprocessing service on the same dataset (instead of trying to run in a feature service), just like I did on my previous posts, and then call this geoprocessing service from the web app after I create the feature. I think it might work. I am going to try that and will keep you posted. Thank you so much for your help again!

Re: Field not get calculated in geoprocessing service

KevinHibma — Sun, 12 Dec 2021 04:46:25 GMT

OR MAYBE THIS, IF YOU LIKE IT BETTER:
(either way, pathnames are assembled out of text passed in)

>>> import os
>>> os.path.join(workspace, GetPolyFC)

'\\\\myServer\\myGDB.sde\\my user-defined value'
>>>

100% use os.path.join.
Please dont "add (+)" paths together. Scripts and publishing of those scripts aren't guaranteed to work as we might not be able to find data. Using the OS methods as you describe is the way to go.

And you're doing great with the explanation.
I tried to help over here: http://gis.stackexchange.com/questions/72719/field-does-not-get-updated-with-geoprocessing-service/72723 but I'm still not 100% sure the goal of this project. Wayne has done a good job explaining the different input methods with GP Server, but I'm still confused if ionara_wilson is wanting to provide features to the GP Service, or if the data is already on the Server.
The Service Editor may modify the input parameter's input mode when publishing based on what we think you want when you have an unsupported input type. Simply pressing the "publish" button without checking and ensuring the input type is correct might be whats going wrong here.

To sum up how it works one more time, because I'm not sure what the desired goal is besides "calculating a value".

--feature class as input is not supported with GP Server. If your model/script is taking a featureclass as input, we change that to FeatureSet in the Service Editor. Your only other option would be to hardcode it by setting it to "constant".
--FeatureSet is a supported input type. If you've set to featureset, we leave it as featureset. FeatureSet allows you to digitize features and send them. If you're using REST, you can send a GPFeatureRecordSet, which is basically just the JSON representation of the features you want the service to use.
--Using a layer as an input the the tool, it gets mapped to Choice List in the Service Editor. A choice list means that and any other layers which are checked off will be available for use in the service. Your "select" them by name when consumed via REST. Looking at that parameter in the REST end point, you'll see that choice list of layers and the name you'd choose.

Based on the last script you've posted, it seems you want to update a featureclass.

At the risk of confusing everything, I've modified the script to make use of LAYERS which get name matched from the tool. (thats how they're found). I suggest this as its a performance gain and the better way to work against data (vs. continually going back and re-opening it at the source: sde). If this doesn't make sense then go the way you are, just change this:
"Database Connections\\Connection to tfsgis-iisd01IWILSON.sde\\%s" %Input_Points
to this:
os.path.join("Database Connections\\Connection to tfsgis-iisd01IWILSON.sde", Input_Points)


import os, sys, arcpy, traceback, arcgisscripting
# DELETED WORKSPACE AND REFERENCE TO SDE FILES

# Script arguments
Selecting_Features = arcpy.GetParameterAsText(0)
value = arcpy.GetParameterAsText(1)
stewardship =  arcpy.GetParameterAsText(2)

# DELETED MAKE FEATURE LAYER....

#Drag the layer from SDE you want into ArcMap. Its this layer:
#Connection to tfsgis-iisd01IWILSON.sde\sars.dbo.CITYTEST
#Name that layer in the TOC "CITYTEST"
#You'll have a variable in the script which references that layer, here:

myLayerToModify = "CITYTEST"
#When you run this script, in ArcMap, it will look for a layer called CITYTEST when you use the myLayerToModify variable. A name matching occurs.

#This variable "myLayerToModify" will be used in the script. You don't need MakeFeatureLayer.

# Process: Select Layer By Location
arcpy.SelectLayerByLocation_management(myLayerToModify, "INTERSECT", Selecting_Features, "", "NEW_SELECTION")


# Process: Calculate Field
#arcpy.CalculateField_management(myLayerToModify, "City, '"' + value + '"', "PYTHON_9.3")

with arcpy.da.UpdateCursor(myLayerToModify, ("City")) as rows:
        # row comes back as a tuple in the order specified here, so Office is row[0], Forester is row[1]
        for row in rows:
            row[0] = value
           
            rows.updateRow(row)

Either approach you take, I don't see any reason why you can't reference the same sde layers here in the GP Service which would also be the same layers the feature service runs off of.
Fast forward to about 28minutes into this video: http://video.esri.com/watch/1658/creating-geoprocessing-services
Scott explain's parameter transformation and datastore, then I do a demo which uses SDE data that is consumed both from a FeatureService and a GP Service. (edits made the to FS impact the result of the GP Service)

Re: Field not get calculated in geoprocessing service

ionarawilson1 — Sun, 12 Dec 2021 04:46:27 GMT

Thank you for the thorough explanation Kevin. The issue I have is that I need to create a web app where users will create a feature and then some values will be calculated, for example an id number that will be called PlanID that will be created from the combination of fiscal year, county where feature is located and a unique number - if there are no other features in that county, the unique number will be 1, so the id number would be for example 2013-230-001. If there is another feature in that county in that year, the id would be 2013-230-002. So I was thinking I could use a feature service for the user to digitize the features in the web app (I already have a web app in javascript with a editor and I can already create features and populate the date field for example). Then I thought I could use a geoprocessing service to calculate the PlanID, based on the year entered and the county where the feature is located, plus that unique value. I already have a python script for the calculation of the PlanID in ArcMap, but I need to change the script to create the geoprocessing service that will be running in the web app after the feature is digitized. Would this be the best approach - creating a feature service to create features in a web app and then run a geoprocessing service to calculate values based on the features created?

So if this works, perhaps I don't even need to select the features in the python script, as they already will be selected when the user is digitizing (editing) the feature in the web app using the editor widget. Would this work?
So far, I have the code that it is updating the year field from the date the user will enter (please see below). I need to have the code where it intersects with the county to get the county number and also define the unique number that goes after the county number.


import os, sys, arcpy, traceback, arcgisscripting
arcpy.env.workspace = "Database Connections\\sars.sde\\"

Input_Points = "Stewardship"


Input_Polygons = "Polygon_Layer"

# Process: Make Feature Layer
arcpy.MakeFeatureLayer_management(Input_Polygons, Input_Polygon_Layer)



with arcpy.da.UpdateCursor(Input_Polygon_Layer, ("DateStart", "FFY")) as rows:


       
    for row in rows:


       Datestarstr1 = row[0]

       Datestarstr2 = str(Datestarstr1)
       
      
       yearonly = Datestarstr2[0:4]

                 
       row[1] = yearonly


       rows.updateRow(row)

So, as you are saying, I should be able to just use a layer (Stewardship). In my case, would the web app be able to identify that layer as the feature class? Please see code below. Thank you!!!


import os, sys, arcpy, traceback, arcgisscripting


Input_Polygons = "Stewardship"


with arcpy.da.UpdateCursor(Input_Polygons, ("DateStart", "FFY")) as rows:


       
    for row in rows:


       Datestarstr1 = row[0]

       Datestarstr2 = str(Datestarstr1)
       
      
       yearonly = Datestarstr2[0:4]

                 
       row[1] = yearonly


       rows.updateRow(row)

Re: Field not get calculated in geoprocessing service

ionarawilson1 — Thu, 17 Oct 2013 16:26:22 GMT

Ok, I found what the problem was. A parameter cannot have a feature class as an input (go figure!), so if you use arcpy.GetParameterAsText() and then have that parameter as a featureset (which I needed for my gp) it won't work. What I did was to create a variable for that parameter and when I run it inside ArcMap I grab the layer for that feature class (in my case the layer is referencing a feature class in a sde geodatabase). Then I publish the gp, then when I run the geoprocessing service I can grab the file directly from the connection instead of the layer. I hope it will help somebody because it took me a while to figure this out.

 Input_Polygons = "Stewardship"    with arcpy.da.UpdateCursor(Input_Polygons, ("DateStart", "PlanID", "FFY")) as rows:      for row in rows:          if not (row[1] or "").strip(): #covers blank, one blank space, or Null             Datestarstr1 = row[0]             Datestarstr2 = str(Datestarstr1)                      yearonly = Datestarstr2[0:4]                               row[2] = yearonly              rows.updateRow(row)