Creating Interactive Maps in GeoAnalytics Engine Using the ArcGIS Maps SDK for JavaScript
GeoAnalytics Engine allows you to plot geometries using st.plot and rt.plot. These generate static map graphics in your PySpark notebook; however, you may want interactivity in your visualizations. Using a Python script and the ArcGIS Maps SDK for JavaScript, you can create embedded interactive maps of both vector and raster layers in your notebook. This blog post shows you how to display complex data in a way that facilitates more dynamic exploration. Using scrolling, zooms, renderers, labels, and popups, data can be explored at a greater depth than is possible with a single image.
To generate interactive maps using the Maps SDK for JavaScript and GeoAnalytics Engine, we have created a python script named geoanalytics_engine_jsmapping.py (we also created a script called geoanalytics_fabric_jsmapping.py if you're working with GeoAnalytics for Microsoft Fabric) that you will need to make accessible in your workspace so that you can reference it in your notebooks. This script can be found in the geoanalytics-engine-samples GitHub repo.
What the script does
The jsmapping script was created by the GeoAnalytics team to provide users of GeoAnalytics Engine with the ability to create interactive maps. The script creates a link between PySpark DataFrames and the ArcGIS Maps SDK for JavaScript that allows the interactive maps to be rendered directly in your notebooks.
The script facilitates interactive mapping with:
- Base maps
- Plotting points, lines, polygons, and rasters
- Symbolization
- Popups
Note that official support for this script is not offered. However, you can file issues and leave comments here or in the GitHub repo; the GeoAnalytics Engine team is open to any feedback you might have.
Set up the workspace
In the following examples, I am using GeoAnalytics Engine in a local Spark environment. I have authorized the API with my user credentials and have loaded the required libraries into my Python environment. Note that EMR notebooks cannot display interactive maps, and Fabric environments have limits on how much data can be displayed on a single map.
Make sure the geoanalytics_engine_jsmapping.py script is downloaded to a location that is accessible to your PySpark notebook. Refer to this guide on how to import a library from a separate directory if needed.
The script includes an EsriJSMap class that displays a default ArcGIS interactive map with an Esri basemap underneath. An EsriJSMap can be initialized without any parameters required. Only the jsmapping script loaded into the workspace and imported into the notebook is required. Let’s look at how this works:
# Import the required modules
import geoanalytics
import geoanalytics.sql.functions as ST
import pyspark.sql.functions as F
# Import and initialize the map
import geoanalytics_engine_jsmapping
map = geoanalytics_engine_jsmapping.EsriJSMap() The feature layer data used in this blog post comes from 2020 census data for the state of Massachusetts. The geometry is at the census tract level. For metadata, view the provided link in the sample below.
# Path to the Massachusetts population data
data_path = "https://services.arcgis.com/P3ePLMYs2RVChkJx/ArcGIS/rest/services/USA_Census_Tracts/FeatureServer/0"
# Create the geometry DataFrame
df = spark.read.format("feature-service").load(data_path)\
.filter(F.col("STATE_ABBR") == "MA")\
.withColumn("shape", ST.transform("shape", 2249))
Visualize data
PySpark DataFrames can be added as either vector layers using the add_layer function or raster layers using the add_raster_layer function.
Before adding any layers, let’s just display the blank map. It should show a light grey basemap of the world fully zoomed out.
# Display the blank map
map.display()
In this example, you can add the DataFrame of all the census tracts of Massachusetts to the EsriJSMap. The display function outputs the interactive map, showing the basic geometry at multiple zoom levels. The EsriJSMap object keeps track of the layers being added, so creating a display with different layers may require redefining the map class before adding new layers. If the same DataFrame is added to the map and the map is displayed before the map is redefined, the layer will be rendered twice unnecessarily. For future examples, this is avoided using the jsmapping script's display_layer and display_raster_layer functions.
# Add and display the DataFrame
map = geoanalytics_engine_jsmapping.EsriJSMap()
map.add_layer(df)
map.display()Add Pop-Ups to data layers
Pop-ups can be added to vector layers as a parameter in the add_layer or display_layer functions. You can provide an array of field names that can appear for each geometry when it is selected. The title of the pop-ups will always be “Fields” unless explicitly stated using a PopupProperties constructor JSON object. In this example, the FIPS code, population, and population per square mile are added as pop-up fields.
For more information about customizing pop-up displays with constructor JSON objects, see the Maps SDK for JavaScript Popup documentation.
# Display the DataFrame with a Pop-Up
map.display_layer(df, popup = ['FIPS', 'POPULATION', 'POP_SQMI'])
Add Labels to data layers
Labels can be defined as the string of a field name or a constructor JSON object of label properties to display over each geometry. Using the label parameter, you can give a quick overview of a field, such as the tract FIPS code.
For more information about customizing labels with constructor JSON objects, visit the Maps SDK for JavaScript Labels documentation.
# Display the DataFrame with a Label
map.display_layer(df, label = "TRACT_FIPS")
Change visuals using Renderers
Renderers can be added to change the symbology of geometries. This can be defined using a constructor JSON object as well, though the jsmapping script offers functions for simple visualization formats.
Different fill and coloring styles exist for multiple geometries. Refer to the style options of simple_marker for point and multipoint geometries, simple_line for line geometries, and simple_fill for polygon geometries.
Using the simple_fill option for polygons, you can change the geometries of a data layer to be solid blue with a white outline.
# Display the DataFrame with a Renderer
renderer = geoanalytics_engine_jsmapping.Renderers
map.display_layer(df, renderer = renderer.simple_fill(color = [55, 83, 140, 0.7], outline = [255, 255, 255, 0.7], style = "solid"))
Display raster data
In addition to adding vector DataFrames, raster DataFrames can also be added to the EsriJSMap. Here, I have used a sample Image Service of AIS passenger vessel positions loaded in as an example raster DataFrame. You can also define your own Colormap or choose from a limited set of pre-made colormaps to display raster data. In this example, the "Precipitation" colormap was used to display the density of passenger vessels from January 2015 to February 2021, where colors of red, yellow, green, and blue represent density from low to high in that order. Please note that Colormaps can only be used as a parameter of the add_raster_layer function.
# Path to the AIS positions reported by passenger vessels
raster_path = "https://tiledimageservices.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services/Global_Ship_Density_Passenger/ImageServer"
# Passenger vessels around the coast of Massachusetts
raster = spark.read.format("image-service").option("extent", "-71.25, 41, -69, 43").load(raster_path)
# Display the raster DataFrame
map = geoanalytics_engine_jsmapping.EsriJSMap()
map.add_raster_layer(raster, colormap = "Precipitation")
map.display()
Create interactive maps with Visual Variables
With Pop-Ups, Labels, and Renderers together, you can create a final interactive map. Using Visual Variables, field values can change the appearance of vector layers. Visual Variables are represented as an item in a renderer JSON object, describing visual changes and the fields that affect them. For this example, a Colormap is used to represent population density. Keep in mind that colormaps that have non-linear or non-categorical styles must be explicitly defined as I have done here. This example combines Popups, Labels, and Renderers to create an interactive map describing the population spread of census tracts in the state of Massachusetts.
# Display the DataFrame using Visual Variables
map.display_layer(df,
label = "TRACT_FIPS",
popup = ['FIPS', 'POPULATION', 'POP_SQMI'],
renderer = {
"type": "simple",
"symbol": {
"type": "simple-fill",
"color": [150, 150, 150, 0],
"outline": {
"color": [255, 255, 255, 0.5],
"width": 0.5,
}
},
"visualVariables": [
{
"type": "color",
"field": "POP_SQMI",
"stops": [
{ "value": 50, "color": "#0d0b16" },
{ "value": 200, "color": "#3b0f6f" },
{ "value": 1000, "color": "#8c2981" },
{ "value": 5000, "color": "#de4968" },
{ "value": 20000, "color": "#fca636" },
{ "value": 100000, "color": "#fbf47a" }
]
}
]
}
)
Conclusion
This article introduces creating JavaScript-enabled maps with GeoAnalytics Engine. While this isn’t an officially maintained feature, we thought it might still be useful for exploring results in your PySpark notebooks. Feel free to reach out to us in the comments section, or in GitHub, to let us know how it works for you and if you find any issues. We’re looking forward to seeing how you use this functionality!