Select to view content in your preferred language

Geo-processing service testing/debugging

3227
13
05-02-2017 10:12 AM
RobertHewlett
Regular Contributor

It would be vital to geo-processing service debugging and testing if the the data formats of "Feature Set" and "Record Set" be properly supported on the Desktop.

If the parameter of the model has the data type set as one of the above (Feature Set/Record Set) then one should be able to load JSON from a text file on disc or point to a feature service in AGO/ArcGIS Enterprise Server via a URL (or exiting connection).

That way, one could get desktop pre-testing closer to the actual environment that will be encountered on the web where JSON feature sets and record sets rule.

 

This is especially important for python script tools where commands such as describe result in objects with very different methods/properties depending on the data format.

Does this not make sense?

13 Replies
RobertHewlett
Regular Contributor

Hi Jonathan,

I will break this into 3 sections:

1. Example Workflow

The generic work flow of "Testing and debugging a new geo-processing service".

  • The model works fine on the desktop (tested with feature classes)
  • Publishes successfully to ArcGIS Enterprise server (10.5)
  • Test the the service by writing a simple web app (ArcGIS JS 4.3)
  • The service throws an error via a tool or the service throws an error via a python script
  •  Why?

If I could test with a true Feature Set before publishing I could eliminate that as an issue and that would speedup the debugging process.

 

2. Example Tools

  • Grid Index Features
  • Recalculate Feature Class Extent
  • Any tool that "cheats with" /requires layer extent information (Feature Class, Feature layer, Shapefiles have extents for the entire dataset)
  • Work around : MBG - tool with the all option
    • This work around is only discovered after burning a lot of time and I would have known immediately if I tested with a true Feature Set

3. Python

  • arcpy.describe
    • Very different properties when you compare Feature Classes and Feature Sets
  • Cursors
    •  arcpy.SearchCusor vs arcpy.da.SearchCursor
      • Work fine on the desktop
      • Blows up on the web
      • Why?
        • On the desktop the Feature Class has a system maintained PK, the OBJECTID
        • On the web, a graphic added via the JS API 4.3 with have, by default, an "ObjectID" of -1 and not unique.
          • In my case the web had two object ids: "OBJECTID" and "ObjectID". "OBJECTID" was an OID and unique but "Object ID" for every feature in the Feature Set was equal to -1
          • JS sees them as different key names because it is case sensitive. Does this break the geo-relational model ... maybe ... maybe not ... depends on the ORDBMS and its case sensitivity.
          • This is where the fun starts with the cursors 
            • arcpy.SearchCusor does not care
            • arcpy.da.SearchCusor self destructs.
      • This issue was discovered and resolved when I sent the input JSON back via arcpy.messages[].
      • I was able to send the JSON back because of the describe command.

I hope that helps.

Rob

0 Kudos
JonathanQuinn
Esri Notable Contributor

Thanks for the breakdown, Robert.

In regards to #1, the best way to debug isn't always by taking it back to the Desktop environment.  Have you enabled INFO or at least WARNING/SEVERE messages on the GP task?  Is it a Python error, or GP runtime error?  Have you tried this in an OOTB application with an OOTB widget like the WAB and GP widget?

For #2, I don't work much with JS, but a geoprocessing feature set would inherit the spatial extent information that was used to define the template for the feature set.

For the last point, a feature class and a feature set on the web are both JSON based on geometry objects.  This allows GP tasks to work against feature services.  The fields are also going to be defined by the template you set for the feature set.  If the template resides in an FGDB or SDE database, then the fields will honor the requirements of that workspace, (all caps OBJECTID, etc).

Is there a sample script you've worked with that works in DT but fails to run when it's a service?  Using the very simple example script below, I see no problems using feature sets or arcpy.da.SearchCursor:

import arcpy, arcpy.da

inFeatures = arcpy.GetParameter(0)

arcpy.AddMessage("In feature fields:")
fields = arcpy.ListFields(inFeatures)
for field in fields:
    arcpy.AddMessage("  " + field.name,)

arcpy.AddMessage("In feature records:")
with arcpy.da.SearchCursor(inFeatures,["*"]) as cursor:
    for row in cursor:
        arcpy.AddMessage("  " + str(row))

arcpy.Buffer_analysis(inFeatures,"in_memory/outBuffer","50 Meters")

arcpy.AddMessage("Out feature fields:")
fields = arcpy.ListFields("in_memory/outBuffer")
for field in fields:
    arcpy.AddMessage("  " + field.name,)

arcpy.AddMessage("Out feature records:")
with arcpy.da.SearchCursor("in_memory/outBuffer",["*"]) as cursor:
    for row in cursor:
        arcpy.AddMessage("  " + str(row))

arcpy.SetParameter(1,"in_memory/outBuffer")

When run in DT:

When run in the WAB using the out of the box GP widget:

Input feature set:
{"geometryType":"esriGeometryPoint","features":[{"geometry":{"x":-10913414.138825795,"y":5223384.6259502,"spatialReference":{"wkid":102100}}},{"geometry":{"x":-11077295.127469169,"y":4753755.524166201,"spatialReference":{"wkid":102100}}},{"geometry":{"x":-10536732.463436546,"y":4922528.482619826,"spatialReference":{"wkid":102100}}},{"geometry":{"x":-10426663.142705921,"y":5267412.35424245,"spatialReference":{"wkid":102100}}},{"geometry":{"x":-10935428.00297192,"y":5551146.603236948,"spatialReference":{"wkid":102100}}}],"sr":{"wkid":102100}}

Output results:

{"jobId":"jee4650a66f2c4836a7bff7b2370b5bb7","jobStatus":"esriJobSucceeded","results":{"Output":{"paramUrl":"results/Output"}},"inputs":{"Input_Feature":{"paramUrl":"inputs/Input_Feature"}},"messages":[{"type":"esriJobMessageTypeInformative","description":"Executing (Feature Set Buffer): FeatureSetBuffer \"Feature Set\""},{"type":"esriJobMessageTypeInformative","description":"Start Time: Thu May 04 17:03:43 2017"},{"type":"esriJobMessageTypeInformative","description":"Executing (FeatureSetBuffer): FeatureSetBuffer \"Feature Set\""},{"type":"esriJobMessageTypeInformative","description":"Start Time: Thu May 04 17:03:43 2017"},{"type":"esriJobMessageTypeInformative","description":"Running script FeatureSetBuffer..."},{"type":"esriJobMessageTypeInformative","description":"In feature fields:"},{"type":"esriJobMessageTypeInformative","description":"  OBJECTID"},{"type":"esriJobMessageTypeInformative","description":"  SHAPE"},{"type":"esriJobMessageTypeInformative","description":"In feature records:"},{"type":"esriJobMessageTypeInformative","description":"  (1, (-10913414.138825797, 5223384.62595021))"},{"type":"esriJobMessageTypeInformative","description":"  (2, (-11077295.127469167, 4753755.5241662115))"},{"type":"esriJobMessageTypeInformative","description":"  (3, (-10536732.463436548, 4922528.482619822))"},{"type":"esriJobMessageTypeInformative","description":"  (4, (-10426663.142705921, 5267412.3542424515))"},{"type":"esriJobMessageTypeInformative","description":"  (5, (-10935428.002971917, 5551146.603236951))"},{"type":"esriJobMessageTypeInformative","description":"Out feature fields:"},{"type":"esriJobMessageTypeInformative","description":"  OBJECTID"},{"type":"esriJobMessageTypeInformative","description":"  SHAPE"},{"type":"esriJobMessageTypeInformative","description":"  BUFF_DIST"},{"type":"esriJobMessageTypeInformative","description":"  ORIG_FID"},{"type":"esriJobMessageTypeInformative","description":"Out feature records:"},{"type":"esriJobMessageTypeInformative","description":"  (1, (-10913414.1388, 5223384.626000001), 50.0, 1)"},{"type":"esriJobMessageTypeInformative","description":"  (2, (-11077295.1275, 4753755.5242), 50.0, 2)"},{"type":"esriJobMessageTypeInformative","description":"  (3, (-10536732.4634, 4922528.4826000035), 50.0, 3)"},{"type":"esriJobMessageTypeInformative","description":"  (4, (-10426663.1427, 5267412.354199997), 50.0, 4)"},{"type":"esriJobMessageTypeInformative","description":"  (5, (-10935428.003, 5551146.603200003), 50.0, 5)"},{"type":"esriJobMessageTypeInformative","description":"Completed script FeatureSetBuffer..."},{"type":"esriJobMessageTypeInformative","description":"Succeeded at Thu May 04 17:03:45 2017 (Elapsed Time: 1.87 seconds)"},{"type":"esriJobMessageTypeInformative","description":"Succeeded at Thu May 04 17:03:45 2017 (Elapsed Time: 1.89 seconds)"}]}

Of course this is a very simple example and you're likely doing something more complex but the underlying idea is that if it works in Desktop, in nearly all cases it will work exactly the same as a service.

0 Kudos
JonathanQuinn
Esri Notable Contributor

I should add what the script tool parameters looks like:

I've set the input feature to be a feature set and defined the schema as a newly created point feature class.  

When run in ArcMap, it gives me the ability to digitize a feature or browse to an existing feature:

Something that may help is including the original feature(s) in the service, so the input features are populated at Rest:

This does't mean it will only run with those features, but instead there's a default value.  You can run the tool directly from the REST endpoint without needing to update any parameters:

0 Kudos
RobertHewlett
Regular Contributor

Hi,

This is for the post above this one:

re # 1: My SOP is messages on at the info level. Also, I found it faster to use VIM and edit the python live on the server (CentOS 7)

re # 2: The 'inheritance' will not happen in 4.3 as there is currently no where to put the information. For example, in the 3.x version of the JS API points do not have extent info, however, in 4.3 the points do have extents. Maybe in a future version of the JS  API the extent will be added to the Feature Set spec but currently it is not there so it will always be lost. Unless ... there is another composite object in the 4.3 spec that could contain an extent as well as a Feature Set but that object would be something else i.e. not a Feature Set. BTW I hit this a lot when explaining the differences between WKT, EWKT, WKB, EWKB and the Geo-Package header that wraps WKB. Check out the output of the Features to JSON tool.

re #3: The default behavior for a Graphic in the 4.3 version of the JS API is 'ObjectID' with an initial value of -1. I agree that if a feature service was published using feature classes from a FGDB then the PK will definitely be "OBJECTID" but not if we are talking interactively drawn graphics in a graphics layer using the 4.3 JS API.

I can get my services to work but it would be very helpful and a much better the user experience if raw JSON in ESRI-JSON-Feature-Set-format could be consumed by Model Builder in an elegant fashion.

I will post later ... back to work for now.

0 Kudos