Skip navigation
All Places > GIS > Web GIS > ArcGIS Online > Blog > 2020 > March
2020

I've had a longstanding need to visualize, query, and filter features using values from a related table, and came up with the following solution, which works great for my use case.

 

The use case involves managing stormwater catch basin inspection and cleaning (I've simplified the workflow for purposes of this post). The customer wanted field workers to be able to open Collector and quickly see basins that need attention or haven't been inspected/cleaned in over a year, and provide that same information in a dashboard. 

 

It's very easy to set up:

 

  1. Add fields to the feature layer to hold the attributes to bring over from the related table
  2. Grab the most recent record from the related table, and write values from that record over to the feature layer using the ArcGIS API for Python
  3. Put script on PythonAnywhere and set it to run every 60 seconds
  4. Configure web map symbology with a simple Arcade expression to show expired and failed inspections

 

Details on each step above:

 

  1. Self-explanatory. I named the fields "Status" and "LastCleaning".
  2. Wrote the script shown at the end of this post to grab records from the related table from the past day, sort them by time and drop duplicate records related to an asset (in case there were two inspection/cleaning records within the past 24 hours -- for example, a failed inspection on Wednesday afternoon was resolved on Thursday morning), then use a unique identifier ('FacilityID') to update the asset with data from the most recent inspection/cleaning. Shout out to Michael Duncan for help with the date query.
    from arcgis import GIS
    from arcgis.features import FeatureLayer
    import pandas as pd
    from arcgis.features import SpatialDataFrame
    from datetime import datetime, timedelta
    import time

    gis = GIS(f"https://someorg.maps.arcgis.com", 'someuser', 'somepass')

    def update_basins():
        one_day = datetime.today() - timedelta(days=1)
        string_day = one_day.strftime('%Y-%m-%d %H:%M:%S')
        where_query = f"DateInspected >= DATE '{string_day}'"

        catch_basins = gis.content.get('21343f6579b74cf212576e5614db8866')
        catch_basins_lyr = catch_basins.layers[0]
        catch_basins_sdf = SpatialDataFrame.from_layer(catch_basins_lyr)
        catch_basins_fset = catch_basins_lyr.query()
        catch_basins_features = catch_basins_fset.features

        cleanings_url = 'https://services9.arcgis.com/iERASXD4kaw1L6en/arcgis/rest/services/this_is_an_example/FeatureServer/1'
        cleanings_lyr = FeatureLayer(cleanings_url)
        cleanings_sdf = SpatialDataFrame.from_layer(cleanings_lyr)
        cleanings_fset = cleanings_lyr.query(where=where_query, out_fields='DateInspected, FacilityID, Status')
        cleanings_features = cleanings_fset.features

        df=cleanings_sdf.sort_values('DateInspected', ascending=False)
        df=df.drop_duplicates(subset="FacilityID")

        overlap_rows = pd.merge(left = catch_basins_sdf, right = df, how='inner', on = 'FacilityID')
        catch_basin_features = catch_basins_fset.features
        cleaning_updates = cleanings_fset.features
        cleaning_updates.reverse()

        def update(basins, cleanings):
            for FacilityID in overlap_rows['FacilityID']:
                try:
                    basin_feature = [f for f in catch_basin_features if f.attributes['FacilityID'] == FacilityID][0]
                    cleaning_feature = [f for f in cleanings_features if f.attributes['FacilityID'] == FacilityID][0]
                    basin_feature.attributes['LastCleaning'] = cleaning_feature.attributes['DateInspected']
                    basin_feature.attributes['Status'] = cleaning_feature.attributes['Status']
                    catch_basins_lyr.edit_features(updates=[basin_feature])
                    print(f"Updated {basin_feature.attributes['FacilityID']} status to {basin_feature.attributes['Status']}", flush=True)
                except:
                    continue

        update(catch_basins_features, cleaning_updates)

    while True:
        update_basins()
        time.sleep(60)

     

  3. Set up an "Always-On" task on PythonAnywhere to continually run the script. This is a very easy process. Just set up a PythonAnywhere account (the free tier would probably be fine for this application), upload your script file, and add the script on the Tasks tab as an Always-On Task. Now, the script writes the most recent inspection/cleaning record to the catch basins attribute table every 60 seconds:
  4. And lastly, just a simple Arcade expression to symbolize by the status of each basin (Current, Expired, or Needs Attention):
    var present = Now()
    var last_cleaning = $feature.LastCleaning
    var cleaning_age = DateDiff(present, last_cleaning, 'years');

    If (cleaning_age < 1 && $feature.Status == 'CleaningComplete') {
        return "Current"
    } else if ($feature.Status == 'NeedsAttention') {
        return "Needs Attention"
    } else if (cleaning_age > 1) {
        return "Expired"
    } else if (IsEmpty($feature.Status)) {
        return "Record missing or incomplete"
    }

 

I hope this is helpful to someone. Feel free to offer suggestions or ask questions. Hang in there everyone!

A Message from Blake Bilbo, Head of Customer Support

During these times, it’s absolutely critical for all GIS teams to make the best possible use of their time and available resources. Taking advantage of the right tools to ensure objectives and responsibilities are accomplished quickly is imperative. To any ArcGIS Online Administrators, or Administrators you may know: Please be aware that there is a FREE version of Admin Tools for ArcGIS Online on the ArcGIS Marketplace that we encourage everyone to take advantage of, if they are not already doing so.

To put this in perspective, many day-to-day administrative tasks will be streamlined and performed in bulk. Rapid access to items, users, and groups throughout an entire Organization via advanced filtering and sorting is supported. There is also the capability to add tags to users, as well as items and groups for important classification on-the-fly as situations develop, and emailing users in bulk ensures everyone in the Organization can be contacted with important information and updates as needed.

For very fluid environments and response, users can rapidly be invited to groups and items can be shared to those groups in bulk. Admin Tools also supports bulk updates to users’ roles as well as their Esri access. In situations where responsibilities and tasks may frequently shift from one user to another, item ownership can be quickly updated and changed. To protect an Organization’s items, delete protection can be turned on in bulk across all items.

Again, we ask that every ArcGIS Online Administrator be taking advantage of these free tools if they are not already doing so. I am also encouraging anyone coordinating with other ArcGIS Administrators and colleagues to share this message to ensure everyone has access to these tools. For any questions, please reach out to us at connect@geo-jobe.com.

Thank you, best wishes, stay healthy, and stay productive!

 

Most of us have not been through something like COVID-19 before. Many authorities and agencies have turned to ArcGIS Online applications to disseminate crucial COVID-19 related information and gain insight for how to respond. You might have come across the Johns Hopkins dashboard, or the web experience created by World Health Organization.

 

ArcGIS Online can be leveraged as a powerful tool for your organization and community in this difficult situation as well, and the picks for this week are intended to provide a firm foundation for getting started. They provide some newly updated best practices for preparing highly scalable ArcGIS Online web applications, comprehensive resources and crucial steps for mapping and understanding COVID-19, a brand new solution template for collecting and sharing information with the public related to the outbreak, and a living document that has collected and categorized COVID-19 dashboard examples, educational resources, questions, and solutions on GeoNet. 

 

Essential Configurations for Highly Scalable ArcGIS Online Web Applications

 

ArcGIS Online Product Manager Kelly Gerrow published this excellent, interactive blog post on the essential best practices for preparing layers and apps for very high demand. As Kelly writes, ArcGIS Online is built on scalable architecture that has supported millions of views per hour on the Johns Hopkins dashboard. The blog features a checklist where you can literally toggle off each setting for your layers and apps as they are configured, ensuring that you tap into the technology that has facilitated this massive-scale information sharing. As the title suggests, it is truly an essential read. 

 

Additional Resources:

 

 

The Tools to Get Started

 

Esri has been working to make resources available for mapping, analyzing, and understanding the spread of COVID-19, as well as gathering information from and sharing information with your community regarding the pandemic. On Esri's COVID-19 Hub Site, you can learn where to begin, access authoritative data and get updates from national agencies. If you want to create your own website to share information and resources with your community, this blog post is an ultra-comprehensive step-by-step guide for accomplishing this using ArcGIS Hub (as many government agencies have already done). And just yesterday, the ArcGIS Solutions team released Coronavirus Response | ArcGIS Solutions for Local Government - this features 10 applications specifically designed for use by the public and public health staff to "to understand the impact of the coronavirus (COVID-19) and share authoritative information about the pandemic with your community." 

 

Join the Conversation on GeoNet

 

Naturally, some of the conversation about the GIS response to COVID-19 has occurred on GeoNet. Leslie Fountain has curated a compelling, living document with links to data sources, dashboard examples, educational resources, and a collaborative spacewhere unanswered COVID-19 questions are highlighted. Taking a look at this space and seeing if you have experience with one of the questions may be a great way to help out a community from afar. 

 

I hope that the picks provided some guidance on the ways that GIS solutions can assist during these challenging days. Let me know in the comments below if they were helpful or if you have any additional questions. Keep an eye out for This Week's Picks - ArcGIS Enterprise & This Week's Picks - ArcGIS Pro. Thanks for reading - my sincere best wishes to you and yours. See you on Tuesday, 3/31.

 

-Peter

 

----------

 

Previous posts: This Week's Picks - ArcGIS Online: Archive 

Connect with me on LinkedIn and follow me on Twitter

On the evening of March 31 (PDT), we plan to update ArcGIS Online.

 

 

New Tools Coming in Beta

  • ArcGIS Notebooks, Esri's integrated Jupyter notebook experience, will be available in beta with ArcGIS Online 8.1. ArcGIS Notebooks will allow you to combine the ArcGIS Python libraries with the larger open-source Python ecosystem for spatial data management, spatial analysis, and data science. 
  • Build simple, responsive applications with the beta version of Esri's popular minimalist Configurable App. Now it can use ArcGIS API for Javascript 4X, and you can include the new functionality from maps authored in Map Viewer beta.
  • A beta for the next version of ArcGIS Dashboards, built using ArcGIS API for Javascript 4x, will be available shortly after the March ArcGIS Online release. New enhancements include improved usability and performance, plus ArcGIS Arcade support for list and indicator elements.

 

HTTP Layer Notifications

ArcGIS Online is preparing to enforce HTTPS-only communication starting September 15, 2020. At the March release, organizations who have chosen not to utilize HTTPS for their web services will receive a notification informing them that adding layers and content via an HTTP URL will not work after September 2020. For more information, read the blog.

 

Regional Subscriptions

ArcGIS Online is launching the capability to choose the region where your geospatial data is stored. With the March update of ArcGIS Online, customers will be able to select their data hosting region (United States or Europe) at the time of purchase. Subscriber and subscription information, item metadata, LBS services and processing (analytics, geocoding, routing, etc.), and back-office systems (subscriber management, credit accounting) will always remain in the US. Regional hosting will be for new subscriptions only.

 

Support for Social Logins with GitHub

Social logins can be an efficient way for organization members to create and manage access to an ArcGIS Online organization account. With the March update of ArcGIS Online, GitHub will be added as a supported social login. 

 

 

Please note that all planned updates are subject to change. A comprehensive list of implemented changes will be provided in our What's New email, blog, and documentation.

 

When an ArcGIS Online update is slated for the near future, I tend to reflect on functionality that has been recently added to the platform. With an update planned for later in March (see this post to participate in the Beta), I realized that there have been a couple of key updates to vector tile layers within the last year. In case you are new to ArcGIS Online within the last couple of months or weren't using vector tiles when the updates went out, the picks for this week showcase some of my go-to vector tile layer resources. I hope you enjoy!

 

Vector Tile Layer Style Editor (Native in Map Viewer)

 

This feature was released in June 2019. Without leaving the Map Viewer, create and save your very own custom style. Zara Matheson's excellent blog will walk you through exactly how: Edit Vector Tiles in Style from Map Viewer 

 

Additional Resources:

 

 

Replace Vector Tile Layers

 

Replacing a vector tile layer with updated data, from the Graphic User Interface (GUI), with minimal downtime, came to ArcGIS Online in October 2019. Garima Tiwari's blog describes in a succinct fashion how to accomplish this: Update hosted vector tile layer content using Replace Layer 

 

Additional Resources:

 

 

Understand the Root Tile

 

When creating vector tile packages in local tiling schemes, it is important to understand the root tile. This Esri Support knowledge base article, written by senior analyst Adam Eversole, sums it up nicely then goes deeper: "If a tile cache were a pyramid, the root tile would be its peak. It is the tile that represents the largest area of the map and the lowest level of detail." This is required reading for those wanting to publish hosted vector tile layers in a local projection. 

 

Additional Resources:

 

 

I hope the above picks provided some guidance and inspiration for working with vector tile layers! Let me know in the comments below if there are any questions, comments, or if there is any content you want to see in the future. Don't forget to keep an eye out for installments of This Week's Picks - ArcGIS Enterprise & This Week's Picks - ArcGIS Pro, coming later this week. Thanks for reading and see you again on 3/17. 

 

Cheers,

 

-Peter

 

--------------------

 

Previous posts: This Week's Picks - ArcGIS Online: Archive 

Connect with me on LinkedIn and follow me on Twitter