Latest Contributions by MarkCederholm

I maintain a number of automated map products in ArcMap which involve not just spatial queries and geometric operations, but also fine-grained manipulation of layers, including renderers and symbology. Let's face it: I never could get the arcpy.mapping module or early versions of ArcGIS Pro to cut the mustard. Later versions of the ArcGIS Pro SDK introduced far greater capability to manipulate map layers and layout elements. But then I asked myself: should users be running Pro at all to create those plots? At Pro 2.4.3, I started taking a closer look at arcpy.mp, wondering if I could create a geoprocessing tool and publish it to a web tool for consumption by a custom Web AppBuilder widget in Portal. I am happy to say that an initial proof-of-concept experiment has been a success. Before I go into that, first I would like to point out some of the features of arcpy.mp that made me decide that it has finally reached the level of functionality that I need: Load and modify symbols Change and manipulate renderers Make layout elements visible or invisible Make modifications at the CIM level One thing arcpy.mp doesn't do yet is create new layout elements, but for my purposes I can recycle existing ones. A good approach is to have a number of elements present for various tasks in a layout, and make them visible or invisible on demand for different situations.     # Show or hide legend legend = self.__layout.listElements("LEGEND_ELEMENT")[0] if self.__bOverview: if self.__bMainline: legend.visible = True else: legend.visible = False else: legend.visible = True ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍     The ability to manipulate legend elements is still pretty limited, but I haven't run into any deal-killers yet. If you really hit a wall, one powerful thing you can now do is dive into the layout's CIM (Cartographic Information Model) and make changes directly to that.  Here's an example of modifying a legend element in a layout via the CIM:     aprx = arcpy.mp.ArcGISProject("c:/apps/Maps/LeakSurvey/LeakSurvey.aprx") layout = aprx.listLayouts("Leak Survey Report Maps Template")[0] cim = layout.getDefinition("V2") legend = None for e in cim.elements: if type(e) == arcpy.cim.CIMLegend: legend = e break legend.columns = 2 legend.makeColumnsSameWidth = True layout.setDefinition(cim) ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍     While the CIM spec is formally documented on GitHub, a simpler way to explore the CIM is to check out the ArcGIS Pro API Reference; all objects and properties in the ArcGIS.Core.CIM namespace should be mirrored in Python.   Part One: Creating a Python Toolbox   LeakSurvey.pyt is in the sample code attached to this post. While my initial draft was focused on successfully generating a PDF file, when the time came to test the tool as a service, additional factors came into play: Getting the service to publish successfully at all Returning a usable link to the resulting PDF file Providing a source for valid input parameters Sharing a geoprocessing tool as a package or service is one of the least intuitive, most trippy experiences I've ever had with any Esri product.  The rationale seems to be that you are not publishing a tool, but a vignette. You can't simply put out the tool and say, here it is: you must publish a geoprocessing result. As part of that concept, any resolvable references will cause ArcGIS to attempt to bundle them, or to match them to a registered data store. This is a great way to get the publication process to crash, or lock the published service into Groundhog Day. So, one key to successfully publishing a web tool is to provide a parameter that: Gives the tool a link to resolve data and aprx references, and When left blank, returns a placeholder result that you can use to publish the service. LeakSurvey.pyt does just that. Here's the definition for the "Project Folder" parameter:     param0 = arcpy.Parameter( displayName = "Project Folder", name = "project_folder", datatype = "GPString", parameterType = "Optional", direction = "Input") ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍     When left blank, the tool simply returns "No results" without throwing an error. Otherwise, it points to a shared folder that contains the ArcGIS Pro project and some enterprise GDB connection files. Returning a usable link to an output file involves a bit of a trick.  Consider the definition of the "Result" parameter:     param7 = arcpy.Parameter( displayName = "Result", name = "result", datatype = "GPString", parameterType = "Derived", direction = "Output") ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍     The tool itself creates a path to the output file as follows:     sOutName = self.__sSurveyType + "_" + self.__sSurveyName + "_" + self.__sMapsheet + "_" sOutName += str(uuid.uuid4()) sOutName += ".pdf" sOutName = sOutName.replace(" ", "_") sOutput = os.path.join(arcpy.env.scratchFolder, sOutName) ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍     If that value is sent to the "Result" parameter, what the user will see is the local file path on the server. In order for the service to return a usable url, a return parameter needs to be defined as follows:     param8 = arcpy.Parameter( displayName = "Output PDF", name = "output_pdf", datatype = "DEFile", parameterType = "Derived", direction = "Output") ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍     Traditional tool validation code is somewhat funky when working with a web tool, and I dispense with it. Rather, the tool returns a list of valid values depending on the parameters provided, keeping in mind that I want this service to be consumed by a web app. For example, if you provide the tool with a survey type and leave the survey name blank, it will return a list of the surveys that exist. If you provide a survey type and name and leave the map sheets parameter blank, it will return a list of the map sheets for that survey:     if self.__sSurveyName == "" or self.__sSurveyName == "#" or self.__sSurveyName == None: # Return list of surveys for type return self.__GetSurveysForType() self.__bMainline = self.__sSurveyType == "MAINLINE" or self.__sSurveyType == "TRANSMISSION" self.__Message("Querying map sheets...") bResult = self.__GetMapsheetsForSurvey() if not bResult: return "No leak survey features." if self.__sMapsheets == None or self.__sMapsheets == "#": # Return list of map sheets for survey sResult = "MAPSHEETS|OVERVIEW" for sName in self.__MapSheetNames: sResult += "\t" + sName return sResult ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍     So how's the performance? Not incredibly great, compared to doing the same thing in ArcObjects, but there are things I can do to improve script performance. For example, because every time the tool is run, it must re-query the survey and its map sheets, there is an option to specify multiple sheets, which will be combined into one PDF, to be returned to the calling application. The tool also supports an "ALL" map sheets option, in order to bypass the need to return a list of map sheets for the survey. Nonetheless, arcpy can suffer in comparison to ArcObjects in various tasks [see this post for some revealing comparisons]. On the other hand, the advantages of using arcpy.mp can outweigh the disadvantages when it comes to automating map production. After testing the tool, it's simple matter to create an empty result and publish it to Portal:     For this example, I also enable messages to be returned:     Once in Portal, it's ready to use:     Part Two: Creating and Publishing a Custom Web AppBuilder Widget   As I've mentioned in another post, one reason I like developing in Visual Studio is that I can create and use project templates. I've attached my current Web AppBuilder custom widget template to this post.     I've also attached the code for the widget itself. Because the widget makes multiple calls to the web tool, it needs a way to sort through the returns. In this example, the tool prefixes "SURVEYS|" when returning a list of surveys, and "MAPSHEETS|" when returning a list of map sheets. When a PDF is successfully generated, the "Result" parameter contains "Success."     private onJobComplete(evt: any): void { let info: JobInfo = evt.jobInfo; this._sJobId = info.jobId; this._gp.getResultData(info.jobId, "result"); } private onGetResultDataComplete(evt: any): void { let val: ParameterValue = evt.result; let sName: string = val.paramName; if (sName === "output_pdf") { this.status("Done."); window.open(val.value.url); this._btnGenerate.disabled = false; return; } let sVal: string = val.value; if (this.processSurveyNames(sVal)) return; if (this.processMapSheets(sVal)) return; if (this.processPDF(sVal)) return; this.status(sVal); } private processSurveyNames(sVal: string): boolean { if (sVal.indexOf("SURVEYS|") !== 0) return false; ... private processMapSheets(sVal: string): boolean { if (sVal.indexOf("MAPSHEETS|") !== 0) return false; ... private processPDF(sVal: string): boolean { if (sVal !== "Success.") return false; ... ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍     The widget can be tested and debugged using Web AppBuilder for ArcGIS (Developer Edition):     Publishing widgets to Portal can be tricky: our production Portal sits in a DMZ, and https calls to another server behind the firewall will fail, so widgets must reside on the Portal server. And even though our "Q" Portal sits behind the firewall and can see other servers, it's on a different domain. Thus, if I choose to host "Q" widgets on a different server, I need to configure CORS.  Here's an example of web.config:     <?xml version="1.0" encoding="UTF-8"?> <configuration> <system.webServer> <cors enabled="true" failUnlistedOrigins="true"> <add origin="*" /> <add origin="https://*.uns.com" allowCredentials="true" maxAge="120"> <allowHeaders allowAllRequestedHeaders="true"> <add header="header1" /> <add header="header2" /> </allowHeaders> <allowMethods> <add method="DELETE" /> </allowMethods> <exposeHeaders> <add header="header1" /> <add header="header2" /> </exposeHeaders> </add> <add origin="https://*.unisource.corp" allowCredentials="true" maxAge="120"> <allowHeaders allowAllRequestedHeaders="true"> <add header="header1" /> <add header="header2" /> </allowHeaders> <allowMethods> <add method="DELETE" /> </allowMethods> <exposeHeaders> <add header="header1" /> <add header="header2" /> </exposeHeaders> </add> <add origin="http://*" allowed="false" /> </cors> </system.webServer> </configuration> ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍     The file sits in a virtual web folder called "Widgets" with any widget folders to publish placed under that. When publishing a widget, initially there may be a CORS error:     but reloading the page and trying again should work.     Once the widget is published to Portal, it can be added to a new or existing application, and it's ready to use:       Because generating plot files can be a lengthy process, it may not be useful for the widget to wait for completion. Were I to put this into production, I would probably modify the tool to send plot files to a shared folder (or even a document management service) and send an email notification when it completes or fails. ... View more

[This was to be my user presentation at the 2020 DevSummit, which was cancelled.] Chrome extensions are a fun way to implement functionality that is not normally available to a web client app. Extensions can make cross-domain requests to gather data from a variety of sources, and at the same time can filter out unwanted content. The Chrome API provides a rich suite of tools for focused application development. Obviously, any app that is implemented as a Chrome extension will only run in Chrome. Also, Chrome extensions must be distributed through Chrome Web Store, but that's not necessarily a bad thing, as I will show later. Here are some online resources: What are extensions? Chrome APIs Debugging Extensions Chrome extensions can contain background scripts, content scripts, a UI for saved user options, and so on. The manifest file is what ties it all together: if you've developed custom widgets for Web AppBuilder, you should already be familiar with the concept. Here's an example of manifest.json:   { "name": "Simple Map Example", "version": "1.0", "description": "Build an Extension with TypeScript and the ArcGIS API for JavaScript 4.x!", "manifest_version": 2, "icons": { "128": "images/chrome32.png" }, "browser_action": { "default_popup": "popup.html", "default_icon": { "128": "images/chrome32.png" } }, "options_ui": { "page": "options.html", "open_in_tab": false }, "permissions": [ "storage" ], "content_security_policy": "script-src 'self' https://js.arcgis.com blob:; object-src 'self'" } ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍   One thing that's worth pointing out is the "content_security_policy" entry. This will be different depending on whether you use JSAPI 3.x or 4.x. See this post for more information. Let's use a Visual Studio 2017 project template (attached) to create a simple extension. Because the template uses TypeScript, there are some prerequisites; see this post for more information. First, let's create a blank solution called DevSummitDemo:     Next, add a new project using the ArcGIS4xChromeExtensionTemplate:     Here is the structure of the resulting project:     Building the project compiles the TypeScript source into corresponding JS files.  Extensions can be tested and debugged using Chrome's "Load unpacked" function:     Note that Chrome DevTools will not load TypeScript source maps from within an extension. That's normally not an issue since you can debug the JS files directly. There is a way to debug the TypeScript source, but it involves some extra work. First, set up IIS express to serve up the project folder:     Then, edit the JS files to point to the localhost url:     Now, you can set a breakpoint in a TS file and it will be hit:     The disadvantage of this approach is that you must re-edit the JS files every time you recompile them.   The next demo involves functionality that is available in JSAPI 3.x, but not yet at 4.x. Namely, the ability to grab an image resource and display it as a layer. Here is a web page that displays the latest weather radar imagery:     The latest image is a fixed url, so nothing special needs to be done to reference it. Wouldn't it be cool, however, to display an animated loop of the 10 latest images? But there's a problem. Let's add the LocalRadarLoop demo project code (attached) to the VS2017 solution and look at pageHelper.ts:   export class myApp { public static readonly isExtension: boolean = false; public static readonly latestOnly: boolean = true; } ‍‍‍‍   When isExtension is false, and latestOnly is true, the app behaves like the web page previously shown. Note also this section of extension-only code that must be commented out for the app to run as a normal web page:   // **** BEGIN Extension-only block **** /* if (myApp.isExtension) { let sDefaultCode: string = defaultLocalCode; chrome.storage.local.get({ localRadarCode: sDefaultCode }, (items: any) => { let sCode: string = items.localRadarCode; let sel: HTMLSelectElement = <HTMLSelectElement>document.getElementById("localRadarCode"); sel.value = sCode; this.setRadar(); }); return; } */ // **** END **** ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍   Because the latest set of radar images do not have fixed names, it is necessary to obtain a directory listing to find out what they are. If you set latestOnly to false and run the app, however, you will run into the dreaded CORS policy error:     This is where the power of Chrome extensions comes into play. Set isExtension to true, and uncomment the extension-only code (which enables a saved user option), and load the app as an extension. Now you get the desired animation loop! Note the relevant line in manifest.json which enables the XMLHttpRequest to run without a CORS error:     Now, as I pointed out earlier, Chrome extensions are distributed through Chrome Web Store:     There are some advantages to this. For example, updates are automatically distributed to users. You can also create an "invisible" store entry, or publish only to testers. I find that last feature useful for distributing an extension that I created for my personal use only. Other distribution options do exist, which you can read about at this link. In conclusion, Chrome extensions enable pure client-side functionality that otherwise would not be possible without the aid of web services. Chrome Web Store provides a convenient way to distribute extensions and updates, with public and private options. [The Local Radar Loop  extension is no longer available at Chrome Web Store.] ... View more