topic Re: Geo-processing service testing/debugging in ArcGIS Enterprise Questions

Geo-processing service testing/debugging

RobertHewlett — Tue, 02 May 2017 17:12:47 GMT

It would be vital to geo-processing service debugging and testing if the the data formats of "Feature Set" and "Record Set" be properly supported on the Desktop.

If the parameter of the model has the data type set as one of the above (Feature Set/Record Set) then one should be able to load JSON from a text file on disc or point to a feature service in AGO/ArcGIS Enterprise Server via a URL (or exiting connection).

That way, one could get desktop pre-testing closer to the actual environment that will be encountered on the web where JSON feature sets and record sets rule.

This is especially important for python script tools where commands such as describe result in objects with very different methods/properties depending on the data format.

Does this not make sense?

Re: Geo-processing service testing/debugging

NeilAyres — Tue, 02 May 2017 18:24:55 GMT

Sorry, but I have only dealt with feature sets input.

And this is an important way to get user input in a geo-processing script.

ie the user inputs some points, or draws a polygon, or what ever.

If you want to suck in a geoJson file, or point to an existing feature in any sort of database, that is a different process.

Re: Geo-processing service testing/debugging

RobertHewlett — Tue, 02 May 2017 18:52:53 GMT

On the web when AGO or ArcGIS JS 4.3 API talks to a geo-processing service then features are sent as JSON:

Parameter: fl_out
Data Type: GPFeatureRecordSetLayer

However, when you are building the model on the Desktop (ArcMap/ArcPro) and test that model (before publishing it) the parameter boxes that appears restricts the user to selecting Feature Classes/Layer files even when one sets the data type to "Feature Set".

Then when the model runs the dialog for the parameter only allows Feature classes

The dialog should allow for JSON (text file with a .JSON extension) or a REST-URL to a layer in feature service.

The JSON to Features tool allows one the select a JSON (See below):

Is that more clear?

Re: Geo-processing service testing/debugging

NeilAyres — Tue, 02 May 2017 18:56:28 GMT

You have to give it a schema or template, so that the feature set knows what sort of geometry to expect - Point, Line or Polygon

Re: Geo-processing service testing/debugging

RobertHewlett — Tue, 02 May 2017 19:06:58 GMT

That would be in the JSON

Re: Geo-processing service testing/debugging

RobertHewlett — Tue, 02 May 2017 19:10:02 GMT

Here are the first few lines of the JSON file:

{
"displayFieldName" : "",
"fieldAliases" : {
"OBJECTID" : "OBJECTID"
},
"geometryType" : "esriGeometryPoint",
"spatialReference" : {
"wkid" : 26910,
"latestWkid" : 26910
},
"fields" : [
{
"name" : "OBJECTID",
"type" : "esriFieldTypeOID",
"alias" : "OBJECTID"
}
],

Re: Geo-processing service testing/debugging

RobertHewlett — Tue, 02 May 2017 19:23:16 GMT

To reiterate, the ability to consume JSON via model parameters on the Desktop before publishing would greatly improve the geo-processing as a service experience. This would match the workflow on the web.

Re: Geo-processing service testing/debugging

JonathanQuinn — Wed, 03 May 2017 02:04:14 GMT

What is the GP service supposed to do? Digitize a point and do some processing? A feature set in Desktop is the same as a GPFeatureRecordSetLayer on the web, so once you define the template, you should be able either digitize a point in ArcMap or use an existing feature class as input. Once it's published, a web application will give you the option to draw a point in the application as the input point(s) to the GP task. The WAB also would allow you to provide a service URL as input.

A quick tour of authoring and sharing geoprocessing services—Documentation | ArcGIS Enterprise

A quick tour of using Feature Set and Record Set—Help | ArcGIS for Desktop

Re: Geo-processing service testing/debugging

RobertHewlett — Wed, 03 May 2017 02:40:42 GMT

Hi Jonathan,

The major issue is that a Feature Set and a Feature class are NOT the same thing in the Desktop world (ArcMap/ArcPro)

Many tools will NOT accept a Feature Set in JSON format as input.

Give it a try by converting an existing Feature class to JSON then try to use the JSON as input for the many tools in ArcToolbox. The tools will not accept it.

Look at arcpy.describe

Describe—ArcPy Functions | ArcGIS Desktop

Notice the difference between a Feature Set and a Feature class.

And that's the 'thing'. The model parameters allow one to set the input data type as Feature Set but the parameter dialog that pops on the Desktop will not let you load a Feature Set (serialize as JSON in a text file) as input.

It would be great if it did. Save a lot of time when you are testing and debugging.

So in the Desktop world where you first build the model for a future web service, you cannot actually test with the actual JSON that it will receive in the future. Instead you test it with a Feature Class which does not exist on the web ... yet.

And that is my point. My services work fine with alterations in python scripts but that needlessly burns a lot of time.

If I could develop and test on the Desktop with raw JSON that would help and make sense.

Thanks for listening,

Rob

Re: Geo-processing service testing/debugging

JonathanQuinn — Wed, 03 May 2017 03:26:41 GMT

I'd say the convenient thing about feature sets in Desktop and their associated GPFeatureRecordSetLayer equivalents is that you typically don't need to provide JSON as input on the desktop side, yet the GP task should perform exactly the same in either environment. Typically, I've seen these tasks revolve around digitizing features in the application, which work fine using Feature Sets in Deskop and the equivalent GPFeatureRecordSetLayer on the Server. Can you describe, (no pun intended), a workflow or specific tools that don't work well when going from the desktop environment to the Server environment? Are you mostly interested in the Describe tool?

Re: Geo-processing service testing/debugging

RobertHewlett — Thu, 04 May 2017 17:26:31 GMT

Hi Jonathan,

I will break this into 3 sections:

1. Example Workflow

The generic work flow of "Testing and debugging a new geo-processing service".

The model works fine on the desktop (tested with feature classes)
Publishes successfully to ArcGIS Enterprise server (10.5)
Test the the service by writing a simple web app (ArcGIS JS 4.3)
The service throws an error via a tool or the service throws an error via a python script
Why?

If I could test with a true Feature Set before publishing I could eliminate that as an issue and that would speedup the debugging process.

2. Example Tools

Grid Index Features
Recalculate Feature Class Extent
Any tool that "cheats with" /requires layer extent information (Feature Class, Feature layer, Shapefiles have extents for the entire dataset)
- A Feature Set does not have/maintain an extent
- FeatureSet | API Reference | ArcGIS API for JavaScript 4.3
Work around : MBG - tool with the all option
- This work around is only discovered after burning a lot of time and I would have known immediately if I tested with a true Feature Set

3. Python

arcpy.describe
- Very different properties when you compare Feature Classes and Feature Sets
Cursors
- arcpy.SearchCusor vs arcpy.da.SearchCursor
  - Work fine on the desktop
  - Blows up on the web
  - Why?
    - On the desktop the Feature Class has a system maintained PK, the OBJECTID
    - On the web, a graphic added via the JS API 4.3 with have, by default, an "ObjectID" of -1 and not unique.
      - In my case the web had two object ids: "OBJECTID" and "ObjectID". "OBJECTID" was an OID and unique but "Object ID" for every feature in the Feature Set was equal to -1
      - JS sees them as different key names because it is case sensitive. Does this break the geo-relational model ... maybe ... maybe not ... depends on the ORDBMS and its case sensitivity.
      - This is where the fun starts with the cursors
        arcpy.SearchCusor does not care
        arcpy.da.SearchCusor self destructs.
  - This issue was discovered and resolved when I sent the input JSON back via arcpy.messages[].
  - I was able to send the JSON back because of the describe command.

I hope that helps.

Rob

Re: Geo-processing service testing/debugging

JonathanQuinn — Sat, 11 Dec 2021 18:41:14 GMT

Thanks for the breakdown, Robert.

In regards to #1, the best way to debug isn't always by taking it back to the Desktop environment. Have you enabled INFO or at least WARNING/SEVERE messages on the GP task? Is it a Python error, or GP runtime error? Have you tried this in an OOTB application with an OOTB widget like the WAB and GP widget?

For #2, I don't work much with JS, but a geoprocessing feature set would inherit the spatial extent information that was used to define the template for the feature set.

For the last point, a feature class and a feature set on the web are both JSON based on geometry objects. This allows GP tasks to work against feature services. The fields are also going to be defined by the template you set for the feature set. If the template resides in an FGDB or SDE database, then the fields will honor the requirements of that workspace, (all caps OBJECTID, etc).

Is there a sample script you've worked with that works in DT but fails to run when it's a service? Using the very simple example script below, I see no problems using feature sets or arcpy.da.SearchCursor:

import arcpy, arcpy.da

inFeatures = arcpy.GetParameter(0)

arcpy.AddMessage("In feature fields:")
fields = arcpy.ListFields(inFeatures)
for field in fields:
    arcpy.AddMessage("  " + field.name,)

arcpy.AddMessage("In feature records:")
with arcpy.da.SearchCursor(inFeatures,["*"]) as cursor:
    for row in cursor:
        arcpy.AddMessage("  " + str(row))

arcpy.Buffer_analysis(inFeatures,"in_memory/outBuffer","50 Meters")

arcpy.AddMessage("Out feature fields:")
fields = arcpy.ListFields("in_memory/outBuffer")
for field in fields:
    arcpy.AddMessage("  " + field.name,)

arcpy.AddMessage("Out feature records:")
with arcpy.da.SearchCursor("in_memory/outBuffer",["*"]) as cursor:
    for row in cursor:
        arcpy.AddMessage("  " + str(row))

arcpy.SetParameter(1,"in_memory/outBuffer")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

When run in DT:

When run in the WAB using the out of the box GP widget:

Input feature set:
{"geometryType":"esriGeometryPoint","features":[{"geometry":{"x":-10913414.138825795,"y":5223384.6259502,"spatialReference":{"wkid":102100}}},{"geometry":{"x":-11077295.127469169,"y":4753755.524166201,"spatialReference":{"wkid":102100}}},{"geometry":{"x":-10536732.463436546,"y":4922528.482619826,"spatialReference":{"wkid":102100}}},{"geometry":{"x":-10426663.142705921,"y":5267412.35424245,"spatialReference":{"wkid":102100}}},{"geometry":{"x":-10935428.00297192,"y":5551146.603236948,"spatialReference":{"wkid":102100}}}],"sr":{"wkid":102100}}

Output results:

{"jobId":"jee4650a66f2c4836a7bff7b2370b5bb7","jobStatus":"esriJobSucceeded","results":{"Output":{"paramUrl":"results/Output"}},"inputs":{"Input_Feature":{"paramUrl":"inputs/Input_Feature"}},"messages":[{"type":"esriJobMessageTypeInformative","description":"Executing (Feature Set Buffer): FeatureSetBuffer \"Feature Set\""},{"type":"esriJobMessageTypeInformative","description":"Start Time: Thu May 04 17:03:43 2017"},{"type":"esriJobMessageTypeInformative","description":"Executing (FeatureSetBuffer): FeatureSetBuffer \"Feature Set\""},{"type":"esriJobMessageTypeInformative","description":"Start Time: Thu May 04 17:03:43 2017"},{"type":"esriJobMessageTypeInformative","description":"Running script FeatureSetBuffer..."},{"type":"esriJobMessageTypeInformative","description":"In feature fields:"},{"type":"esriJobMessageTypeInformative","description":" OBJECTID"},{"type":"esriJobMessageTypeInformative","description":" SHAPE"},{"type":"esriJobMessageTypeInformative","description":"In feature records:"},{"type":"esriJobMessageTypeInformative","description":" (1, (-10913414.138825797, 5223384.62595021))"},{"type":"esriJobMessageTypeInformative","description":" (2, (-11077295.127469167, 4753755.5241662115))"},{"type":"esriJobMessageTypeInformative","description":" (3, (-10536732.463436548, 4922528.482619822))"},{"type":"esriJobMessageTypeInformative","description":" (4, (-10426663.142705921, 5267412.3542424515))"},{"type":"esriJobMessageTypeInformative","description":" (5, (-10935428.002971917, 5551146.603236951))"},{"type":"esriJobMessageTypeInformative","description":"Out feature fields:"},{"type":"esriJobMessageTypeInformative","description":" OBJECTID"},{"type":"esriJobMessageTypeInformative","description":" SHAPE"},{"type":"esriJobMessageTypeInformative","description":" BUFF_DIST"},{"type":"esriJobMessageTypeInformative","description":" ORIG_FID"},{"type":"esriJobMessageTypeInformative","description":"Out feature records:"},{"type":"esriJobMessageTypeInformative","description":" (1, (-10913414.1388, 5223384.626000001), 50.0, 1)"},{"type":"esriJobMessageTypeInformative","description":" (2, (-11077295.1275, 4753755.5242), 50.0, 2)"},{"type":"esriJobMessageTypeInformative","description":" (3, (-10536732.4634, 4922528.4826000035), 50.0, 3)"},{"type":"esriJobMessageTypeInformative","description":" (4, (-10426663.1427, 5267412.354199997), 50.0, 4)"},{"type":"esriJobMessageTypeInformative","description":" (5, (-10935428.003, 5551146.603200003), 50.0, 5)"},{"type":"esriJobMessageTypeInformative","description":"Completed script FeatureSetBuffer..."},{"type":"esriJobMessageTypeInformative","description":"Succeeded at Thu May 04 17:03:45 2017 (Elapsed Time: 1.87 seconds)"},{"type":"esriJobMessageTypeInformative","description":"Succeeded at Thu May 04 17:03:45 2017 (Elapsed Time: 1.89 seconds)"}]}

Of course this is a very simple example and you're likely doing something more complex but the underlying idea is that if it works in Desktop, in nearly all cases it will work exactly the same as a service.

Re: Geo-processing service testing/debugging

JonathanQuinn — Fri, 05 May 2017 16:50:59 GMT

I should add what the script tool parameters looks like:

I've set the input feature to be a feature set and defined the schema as a newly created point feature class.

When run in ArcMap, it gives me the ability to digitize a feature or browse to an existing feature:

Something that may help is including the original feature(s) in the service, so the input features are populated at Rest:

This does't mean it will only run with those features, but instead there's a default value. You can run the tool directly from the REST endpoint without needing to update any parameters:

Re: Geo-processing service testing/debugging

RobertHewlett — Fri, 05 May 2017 19:48:55 GMT

Hi,

This is for the post above this one:

re # 1: My SOP is messages on at the info level. Also, I found it faster to use VIM and edit the python live on the server (CentOS 7)

re # 2: The 'inheritance' will not happen in 4.3 as there is currently no where to put the information. For example, in the 3.x version of the JS API points do not have extent info, however, in 4.3 the points do have extents. Maybe in a future version of the JS API the extent will be added to the Feature Set spec but currently it is not there so it will always be lost. Unless ... there is another composite object in the 4.3 spec that could contain an extent as well as a Feature Set but that object would be something else i.e. not a Feature Set. BTW I hit this a lot when explaining the differences between WKT, EWKT, WKB, EWKB and the Geo-Package header that wraps WKB. Check out the output of the Features to JSON tool.

re #3: The default behavior for a Graphic in the 4.3 version of the JS API is 'ObjectID' with an initial value of -1. I agree that if a feature service was published using feature classes from a FGDB then the PK will definitely be "OBJECTID" but not if we are talking interactively drawn graphics in a graphics layer using the 4.3 JS API.

I can get my services to work but it would be very helpful and a much better the user experience if raw JSON in ESRI-JSON-Feature-Set-format could be consumed by Model Builder in an elegant fashion.

I will post later ... back to work for now.