Migrating a Plot from ArcObjects and ArcMap to arcpy.mp and Portal
I maintain a number of automated map products in ArcMap which involve not just spatial queries and geometric operations, but also fine-grained manipulation of layers, including renderers and symbology. Let's face it: I never could get the arcpy.mapping module or early versions of ArcGIS Pro to cut the mustard. Later versions of the ArcGIS Pro SDK introduced far greater capability to manipulate map layers and layout elements. But then I asked myself: should users be running Pro at all to create those plots?
At Pro 2.4.3, I started taking a closer look at arcpy.mp, wondering if I could create a geoprocessing tool and publish it to a web tool for consumption by a custom Web AppBuilder widget in Portal. I am happy to say that an initial proof-of-concept experiment has been a success.
Before I go into that, first I would like to point out some of the features of arcpy.mp that made me decide that it has finally reached the level of functionality that I need:
- Load and modify symbols
- Change and manipulate renderers
- Make layout elements visible or invisible
- Make modifications at the CIM level
One thing arcpy.mp doesn't do yet is create new layout elements, but for my purposes I can recycle existing ones. A good approach is to have a number of elements present for various tasks in a layout, and make them visible or invisible on demand for different situations.
# Show or hide legend
legend = self.__layout.listElements("LEGEND_ELEMENT")[0]
if self.__bOverview:
if self.__bMainline:
legend.visible = True
else:
legend.visible = False
else:
legend.visible = True
The ability to manipulate legend elements is still pretty limited, but I haven't run into any deal-killers yet. If you really hit a wall, one powerful thing you can now do is dive into the layout's CIM (Cartographic Information Model) and make changes directly to that. Here's an example of modifying a legend element in a layout via the CIM:
aprx = arcpy.mp.ArcGISProject("c:/apps/Maps/LeakSurvey/LeakSurvey.aprx")
layout = aprx.listLayouts("Leak Survey Report Maps Template")[0]
cim = layout.getDefinition("V2")
legend = None
for e in cim.elements:
if type(e) == arcpy.cim.CIMLegend:
legend = e
break
legend.columns = 2
legend.makeColumnsSameWidth = True
layout.setDefinition(cim)
While the CIM spec is formally documented on GitHub, a simpler way to explore the CIM is to check out the ArcGIS Pro API Reference; all objects and properties in the ArcGIS.Core.CIM namespace should be mirrored in Python.
Part One: Creating a Python Toolbox
LeakSurvey.pyt is in the sample code attached to this post. While my initial draft was focused on successfully generating a PDF file, when the time came to test the tool as a service, additional factors came into play:
- Getting the service to publish successfully at all
- Returning a usable link to the resulting PDF file
- Providing a source for valid input parameters
Sharing a geoprocessing tool as a package or service is one of the least intuitive, most trippy experiences I've ever had with any Esri product. The rationale seems to be that you are not publishing a tool, but a vignette. You can't simply put out the tool and say, here it is: you must publish a geoprocessing result. As part of that concept, any resolvable references will cause ArcGIS to attempt to bundle them, or to match them to a registered data store. This is a great way to get the publication process to crash, or lock the published service into Groundhog Day.
So, one key to successfully publishing a web tool is to provide a parameter that:
- Gives the tool a link to resolve data and aprx references, and
- When left blank, returns a placeholder result that you can use to publish the service.
LeakSurvey.pyt does just that. Here's the definition for the "Project Folder" parameter:
param0 = arcpy.Parameter(
displayName = "Project Folder",
name = "project_folder",
datatype = "GPString",
parameterType = "Optional",
direction = "Input")
When left blank, the tool simply returns "No results" without throwing an error. Otherwise, it points to a shared folder that contains the ArcGIS Pro project and some enterprise GDB connection files.
Returning a usable link to an output file involves a bit of a trick. Consider the definition of the "Result" parameter:
param7 = arcpy.Parameter(
displayName = "Result",
name = "result",
datatype = "GPString",
parameterType = "Derived",
direction = "Output")
The tool itself creates a path to the output file as follows:
sOutName = self.__sSurveyType + "_" + self.__sSurveyName + "_" + self.__sMapsheet + "_"
sOutName += str(uuid.uuid4())
sOutName += ".pdf"
sOutName = sOutName.replace(" ", "_")
sOutput = os.path.join(arcpy.env.scratchFolder, sOutName)
If that value is sent to the "Result" parameter, what the user will see is the local file path on the server. In order for the service to return a usable url, a return parameter needs to be defined as follows:
param8 = arcpy.Parameter(
displayName = "Output PDF",
name = "output_pdf",
datatype = "DEFile",
parameterType = "Derived",
direction = "Output")
Traditional tool validation code is somewhat funky when working with a web tool, and I dispense with it. Rather, the tool returns a list of valid values depending on the parameters provided, keeping in mind that I want this service to be consumed by a web app. For example, if you provide the tool with a survey type and leave the survey name blank, it will return a list of the surveys that exist. If you provide a survey type and name and leave the map sheets parameter blank, it will return a list of the map sheets for that survey:
if self.__sSurveyName == "" or self.__sSurveyName == "#" or self.__sSurveyName == None:
# Return list of surveys for type
return self.__GetSurveysForType()
self.__bMainline = self.__sSurveyType == "MAINLINE" or self.__sSurveyType == "TRANSMISSION"
self.__Message("Querying map sheets...")
bResult = self.__GetMapsheetsForSurvey()
if not bResult:
return "No leak survey features."
if self.__sMapsheets == None or self.__sMapsheets == "#":
# Return list of map sheets for survey
sResult = "MAPSHEETS|OVERVIEW"
for sName in self.__MapSheetNames:
sResult += "\t" + sName
return sResult
So how's the performance? Not incredibly great, compared to doing the same thing in ArcObjects, but there are things I can do to improve script performance. For example, because every time the tool is run, it must re-query the survey and its map sheets, there is an option to specify multiple sheets, which will be combined into one PDF, to be returned to the calling application. The tool also supports an "ALL" map sheets option, in order to bypass the need to return a list of map sheets for the survey.
Nonetheless, arcpy can suffer in comparison to ArcObjects in various tasks [see this post for some revealing comparisons]. On the other hand, the advantages of using arcpy.mp can outweigh the disadvantages when it comes to automating map production.
After testing the tool, it's simple matter to create an empty result and publish it to Portal:
For this example, I also enable messages to be returned:
Once in Portal, it's ready to use:
Part Two: Creating and Publishing a Custom Web AppBuilder Widget
As I've mentioned in another post, one reason I like developing in Visual Studio is that I can create and use project templates. I've attached my current Web AppBuilder custom widget template to this post.
I've also attached the code for the widget itself. Because the widget makes multiple calls to the web tool, it needs a way to sort through the returns. In this example, the tool prefixes "SURVEYS|" when returning a list of surveys, and "MAPSHEETS|" when returning a list of map sheets. When a PDF is successfully generated, the "Result" parameter contains "Success."
private onJobComplete(evt: any): void {
let info: JobInfo = evt.jobInfo;
this._sJobId = info.jobId;
this._gp.getResultData(info.jobId, "result");
}
private onGetResultDataComplete(evt: any): void {
let val: ParameterValue = evt.result;
let sName: string = val.paramName;
if (sName === "output_pdf") {
this.status("Done.");
window.open(val.value.url);
this._btnGenerate.disabled = false;
return;
}
let sVal: string = val.value;
if (this.processSurveyNames(sVal))
return;
if (this.processMapSheets(sVal))
return;
if (this.processPDF(sVal))
return;
this.status(sVal);
}
private processSurveyNames(sVal: string): boolean {
if (sVal.indexOf("SURVEYS|") !== 0)
return false;
...
private processMapSheets(sVal: string): boolean {
if (sVal.indexOf("MAPSHEETS|") !== 0)
return false;
...
private processPDF(sVal: string): boolean {
if (sVal !== "Success.")
return false;
...
The widget can be tested and debugged using Web AppBuilder for ArcGIS (Developer Edition):
Publishing widgets to Portal can be tricky: our production Portal sits in a DMZ, and https calls to another server behind the firewall will fail, so widgets must reside on the Portal server. And even though our "Q" Portal sits behind the firewall and can see other servers, it's on a different domain. Thus, if I choose to host "Q" widgets on a different server, I need to configure CORS. Here's an example of web.config:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<cors enabled="true" failUnlistedOrigins="true">
<add origin="*" />
<add origin="https://*.uns.com"
allowCredentials="true"
maxAge="120">
<allowHeaders allowAllRequestedHeaders="true">
<add header="header1" />
<add header="header2" />
</allowHeaders>
<allowMethods>
<add method="DELETE" />
</allowMethods>
<exposeHeaders>
<add header="header1" />
<add header="header2" />
</exposeHeaders>
</add>
<add origin="https://*.unisource.corp"
allowCredentials="true"
maxAge="120">
<allowHeaders allowAllRequestedHeaders="true">
<add header="header1" />
<add header="header2" />
</allowHeaders>
<allowMethods>
<add method="DELETE" />
</allowMethods>
<exposeHeaders>
<add header="header1" />
<add header="header2" />
</exposeHeaders>
</add>
<add origin="http://*" allowed="false" />
</cors>
</system.webServer>
</configuration>
The file sits in a virtual web folder called "Widgets" with any widget folders to publish placed under that. When publishing a widget, initially there may be a CORS error:
but reloading the page and trying again should work.
Once the widget is published to Portal, it can be added to a new or existing application, and it's ready to use:
Because generating plot files can be a lengthy process, it may not be useful for the widget to wait for completion. Were I to put this into production, I would probably modify the tool to send plot files to a shared folder (or even a document management service) and send an email notification when it completes or fails.