Skip navigation
All Places > Applications Prototype Lab > Blog > Authors bgerlt-esristaff

I recently updated the Distributive Flow Lines (DFL) tool, simplifying the interface and making it compatible with Pro.  If you want to use the tool in ArcMap, the previous version of the DFL and the previous blog are still available.  The previous blog also contains a little background on distributive flow maps and some details about the internal workings of the tool.  Here I will focus on how to use the new Pro tool and a couple details about the inputs and flow “direction”.  The example in this blog shows the flow of federal education tax dollars from Washington D.C. to the lower 48 state capitals.  If you would like to follow along, the tool and test data used to produce the maps in this blog are available at the first link above. 

Note: To use the tool you need ArcGIS Pro and the Spatial Analyst extension.  If you do not have access to a Spatial Analyst license, a 30-day no cost trial can be obtained through the Esri Marketplace.

Usually, flow maps depict the flow of something from a single source to many destinations. They can also show stuff flowing from many destinations to a single source.  The DFL tool can be used for both cases.   Within the interface the point of convergence is named Source Feature.   Behind the scenes the “something” always flows from the Destinations to the Source.  This is because the tool uses ArcGIS hydrology GP tools and the flow lines are more akin to a stream network with the mouth of the largest stream terminating at the Source node.  The Source Feature is just the location where the flow lines will terminate and does not need to have any specific fields describing the “something” flowing through the network.

Figure 1: New Distributive Flow Lines Tool

Figure 1: New Distributive Flow Lines Tool

 

The Destination Features in Figure 1 must have an integer field indicating the amount of “stuff” received from the Source.  In Figure 1, the Source Feature, DC Point, is a point feature over Washington DC.  StateCaps represents the lower 48 state capitals.  Edu_Dollars is a field in the StateCaps feature class representing an federal education tax dollars supplied to the states. Figure 2, below, is the output generated based on the inputs in Figure 1.

Figure 2: Output based on Figure1 input values. California and Nevada flow southward to avoid the red barrier.

Figure 2: Output based on Figure1 input values. California and Nevada flow southward to avoid the red barrier.

 

In previous versions of the DFL, the optional inputs, Impassable Features and Impedance Features, also caused some confusion because they are similar but treated much differently by the tool. Both provide some control over where the flow lines will be placed. In Figure 2, the large red line in the western half of the US is the Impassable Features input.  The blue buffers around the capitals are the Impedance Features input.  Impassable features will not be crossed or touched by flow lines.  In addition, they will be slightly buffered and the lines will appear to flow around them.  The Impedance Features may be crossed by flow lines but in most cases the tool will also avoid these features unless there is no other less “expensive” path toward the Source Feature.  Figure 3 represents the output where no Impassable Features are specified. Note the flow lines from California and Nevada change from southward to northward.

Figure 3:  Impedance Features input specified but no Impassable Feature input. Now flow lines generally go around the intermediate state capitals.

Figure 3:  Impedance Features input specified but no Impassable Feature input. Now flowlines generally go around the intermediate state capitals.

 

In Figure 4 below, neither Impassable nor Impedance features were specified.  As you can see, flow lines pass through the intermediate state capitals.  This is sometimes desired, but in the case of federal tax dollars, the dollars do not flow through intermediate states, so this might be confusing.  Providing an Impedance feature reduces this confusion.  If the buffers around the state capitals were specified as Impassable Features, the flow lines could not flow away from the states and no solution would be possible. 

Figure 4: Output generated without specifying Impassable or Impedance Features.  California and Nevada flow northward. Flow lines flow through intermediate state capitals

Figure 4: Output generated without specifying Impassable or Impedance Features.  California and Nevada flow northward. Flow lines flow through intermediate state capitals

 

The output in Figure 5 below used the same inputs as Figure 4 except the “Split flow lines close to” parameter was changed from Destination to Source. The result is that California has a dedicated line all the way into Missouri, and several things change in the Northeast.  This may be less aesthetically pleasing but does a better job of highlighting which individual states receive more tax dollars.

Figure 5: Split flow lines close to Destinations, Neither Impedance nor Impassable features specified.

Figure 5: Split flow lines close to Destinations, Neither Impedance nor Impassable features specified.

 

Figure 6 is a closeup of what is going on in the Northeast. There are a few things worth pointing out. The first is the treatment of the Impedance Features, StateCaps_Buffer.  Notice how the flow lines pass through the New York and Connecticut buffer features. This is happening because the direct route is less “expensive” than going around those buffers.  Purple labels indicate where the values on the flow lines are coming from. The green flow line labels emphasize the additive nature when individual tributary flow lines converge as they get closer to the Source feature.  Lastly, the Massachusetts flow line goes directly through Rhode Island. This is because it is located within the Rhode Island StateCaps_Buffer. This is a case where some manual editing may be needed to clarify that Massachusetts tax dollars are not flowing through Rhode Island.

Figure 6: Note the flow line pass through the buffers around New York and Connecticut as well as Rhode Island. Also note the additive nature of the flow lines.

Figure 6: Note the flow line pass through the buffers around New York and Connecticut as well as Rhode Island. Also note the additive nature of the flow lines.

 

I hope you will find the tool useful in creating flow maps or other creative applications.  I also look forward to reading your comments, suggestions and use cases.  If you missed the link to the tool and sample data, here it is again.  Distributive Flow Lines for Pro.

The graph theory concept of Centrality has gained popularity in recent years as a way to gain insight into network behavior. In graph or network theory, Centrality measures are used to determine the relative importance of a vertex or edge within the overall network. There are many types of centrality. Betweenness centrality measures how often a node or edge lies along the optimum path between all other nodes in the network. A high betweenness centrality value indicates a critical role in network connectivity. Because there are currently no Centrality tools in ArcGIS, I created a simple ArcGIS Pro 2.1 GP toolbox that uses the NetworkX Python library to make these types of analyses easy to incorporate in ArcGIS workflows.  

 Centrality Analysis Tools

  Figure 1 Centrality Analysis Tools (CAT)

The terms network and graph will be used interchangeably in this blog. Here, network does not refer to an ArcGIS Network dataset. It simply means a set of node objects connected by edges. In ArcGIS these nodes might be points, polygons or maybe even other lines. The edges can be thought of as the polylines that connect two nodes. The network could also be raster regions connected by polylines traversing a cost surface using Cost Connectivity.

 

             

Figure 2 A few mid-western urban areas connected by major roads. Cost Connectivity was used to find

"natural" neighbors and connect the towns via the road network.

 

As it turns out, the output from Cost Connectivity (CC) is perfect input for the Centrality Analysis tools. Let’s take a look at the CC output table.

Figure 3 Cost Connectivity output with "out_neighbors_paths" option selected.

 

Now let’s see how this lines up with CAT Node Centrality input parameters.

 

 Figure 4 Node Centrality tool parameters 

 

There are a couple things worth mentioning here. The Starting Node Field and Ending Node Field do not indicate directionality. In fact, the tool assumes cost is the same to move in either direction. I used Shape_Length but could have used PathCost or some other field indicating the cost to move from node to node. This table and its associated feature class are created by Cost Connectivity when you select the “out_neighbor_paths” option. While the minimum spanning tree option will work, the Neighbor output seems more reasonable for centrality analysis. It is also important to make sure you do not have links in your graph that connect a node to itself and that all link costs are greater than zero.

 

Figure 5 Options for Node Centrality type

 

Both the Node and Edge Centrality tools require “connected” graphs, which means all the nodes in the graph must be connected to the rest of the network. If you have nodes that are not connected or reachable by all the other nodes, some functions will not work. This can happen when you have nodes on islands that are unreachable for some reason. If this happens, you will have to either make a connection and give it a really high cost or remove those nodes from the analysis.

 

Because these tools require some specific input, I included a Graph Info tool so that users could get information about the size and connectedness of their input data before trying to run either the Node or Edge centrality tools.

 

Figure 6 Graph Info tool provides critical information about potential input

data without having to run one of the tools first.

 

One last thing to keep in mind -- many of the centrality measures available within these tools require the optimum path between all nodes in the network to be calculated. This is quite compute intensive, and execution time and computer resource requirements grow exponentially. It is best to try the tool out on a fairly small network of 1000 nodes and maybe 5000 connectors before trying to run on larger datasets, just to get a feel for time and resource requirements. The example shown above runs in less than five seconds but there are only 587 nodes and 1469 connectors.

 

Please download the toolbox, try it out, and let me know what you think.  I would like to hear about your use cases.

This Spring we released a new version of the 3D Fences python tool box.  The new release works in both ArcGIS Pro and ArcMap. The 3D Fences Toolbox was introduced in November 2015 and originally ran only in ArcMap.  For detailed description of the toolbox and how it works please read Transforming 3D Data into Fences and Curtains with Geostatistical Tools,  ArcUser Winter 2016 article. 

This python toolbox now consists of two tools enabling the interpolation of point data in the Z (vertical) dimension.   The Parallel Fences tool creates equally spaced fences parallel to the X, Y or Z dimension. X and Y dimension fences will run East-West and North-South respectively in the Z dimension.   Z dimension fences will  be stacked in the standard XY "flat" (horizontal) planes at regular Z intervals.  The Feature-Based Fences tool creates vertical fences in the Z dimension along features in a polyline feature class and,  in Pro, it can also use Map Notes line layers as input.   The Map Notes line layers enable users to create ad hoc features along which fences will be interpolated.  An added benefit of using the toolbox within Pro is that it can be used with either a Scene or Map.  The primary output is a 3D point feature class, optionally time enabled.

The original toolbox also contained a third Interactive Fence tool that only works in ArcMap.  This tool is not needed in Pro because use of Map Notes features provides a new way to achieve the same functionality using the Feature-Based fence tool with better control of fence location, creation, selection and symbology.

 

Using Map Notes in an ArcGIS Pro Scene to create 3D fences

Insert either a Scene or Map in your Pro project and add your point data to the TOC. (I am working in a scene below. My TestPts layer represents a fictional oil leak in the Gulf of Mexico.)

Insert Map Notes, and remove point and polygon map note layers to avoid confusion.

Open the Edit tab, select Create Features, and then select your Map Notes line feature in the Create Features pane.

Digitize line features in you Map or Scene and save Edits.

Run the 3D Fences, Feature-Based Fences tool.

Style your output layers and view.

 

In the screenshot above, I symbolized the points used in the interpolation as red dots.  You can also see the end of the Map Notes base fence protruding past the interpolated fence.  This is because the tool limits interpolation to the extent of the data, not the fence.  I would be interested in hearing your opinions about enabling  extrapolation to the full length of the fence and limiting the results to the minimum bounding area of the sample points instead of the minimum bounding envelope of the sample points. 

All feedback is welcome and I hope you find the toolbox helpful.

12.png

 

NOTE March, 2019:  An even newer and easier to use version of the this tool is now available for ArcGIS Pro and an updated blog post about using the tool has also been posted.

 

NOTE:  The Distributive Flow Lines tool was updated  March 1, 2017.  Please download the latest version.  The interface has changed a little since the original post so some of the description below will be a little off but the concepts are still very close so I have left the graphics and captions alone for now.

 

Based on comments and feedback from the first version of the Flow Map tool and blog post we gave it a second look and have released an update.  As you can see in the screen capture above, the output is similar to the original tool but we have switched to an all raster approach which provides more control over the lines and reduces processing time.

21.jpg

A distributive flow map made by Charles Joseph Minard showing coal exports from England in 1864.

 

 

I have pasted the introductory text from Brad Simantel’s original post below to provide a concise introduction to the history and use of Flow Maps.

“Flow maps are used to show the movement of goods or people from one place to another. These maps use lines to symbolize the movement, often varied in width to represent the quantity of the flow, and fall into one of three categories: radial, network, and distributive. Radial flow maps are used to show relationships between one source and many destinations. If there are more than a handful of destinations and you still want to show the quantity of flow, however, the lines overlap too much to discern individual values. Network flow maps are used to show the quantity of flow over some existing network — transportation and communication networks being the most common. Distributive flow maps are similar to radial flow maps, but rather than having individual lines from the source to each destination, lines are joined together, only forking once they get close to their destinations”.


If you would like to try it out as you read this post, you can access the new  Distributive Flow Lines tool (DFLT) on ArcGIS Online. If you do not have data handy, the English Coal export data, used to create the first map in this post, is also available for download on ArcGIS Online.Tip: The DFLT requires the Distributed Quantity Field to be of type Integer.  If you use the British Coal sample data a new integer field will need to be added and calculated based on the existing CoalTonnage field.  The current field represents thousands of tons so multiplying by 10 and rounding to the nearest integer would represent hundreds of tons rather than thousands.

 

Like the first Flow Map tool, the Spatial Analyst extension is required to use the new DFLT.   In fact, the new tool is completely raster based until the end when the flow line feature class is created as output.

 

The DFLT will accept polygon or point features as input for the Source and Destinations, however, point features are recommended.   If polygons feature classes are used as input for either Source or Destinations they will be converted to points for internal use but these intermediate point datasets are not saved as part of the tool output.  After the Destination feature class is selected the user will need to select an integer field from this dataset representing the quantity that will flow to the destinations from the source.

 

The DFLT provides an option for the user to specify a feature class to use as impassable features.  These features will be buffered by 1.4 times the Cell Size parameter.  The area inside this buffer will be NoData and will be used as a processing mask within the tool.  A second, slightly larger buffer, 3 times the Cell Size parameter, is also created. The area between the inside buffer and this larger buffer is given a very high cost so Flow Direction will not calculate routes that go into the NoData area (most of the time).

 

The second optional parameter is a Impedance feature class.  These polygons represent  features in your map that you would like flow lines to avoid as much as possible.  How much these features are avoided is controlled by the Impedance Weight slider.  When impedance features are specified, they are given a high cost; but may still be crossed in extreme cases where there is no alternative or where they can be crossed in a narrow location as shown in the example below.Tip: One common effect of the DFLT is that it will often skirt Impedance features too close.  This can be avoided by buffering the Foreground features and then using this new feature class as the Foreground features in the tool.

 

Since the primary use case for the DFLT is to create pleasing flow lines, the user will most likely run it several times using different values for the weight sliders to achieve a good starting point for the effect they are looking for.  Using an all raster approach makes this process a bit easier and faster.  In raster processing, cell size has a major effect on processing time. The DFLT calculates a default cell that size works well for the final product.  In most cases it will not be necessary to use a smaller cell size than the default. We recommend using a much larger cell size during initial iterations to reduce processing time until you start getting results close to what you want.toolmenu.jpgTip: It is a good idea to make a note of the default cell size for final processing.  For initial runs of the tool,  multiplying the default cell size by 5 or 10  works well to get a feel for how the various weights and optional parameters effect the output flow lines.  More drastic changes in cell size may have a significant effect on how the lines are routed around Impassable and Foreground features.Tip: If you use the post 10.1 sp1 version of the tool you will notice the tool dialog has been simplified.  Source Weight and Destination Weight sliders have been replaced by a single slider to control how close to the Source or Destinations the flow lines branch.  Also processing extent options have been reduced to "Same as Display" and "Same as Destinations plus Source."  This was changed to prevent most of the cases where no solution is possible. 

 

Cell size is so important in the DFLT because the tool calculates Euclidean distances from source and from all of the destination features. These two distance rasters are then Sliced into an equal number of discrete cost classes.  The number of classes is determined by dividing the maximum processing extent dimension by the cell size.  This acts to normalize the effect of the Source and Destination cost rasters and provide a basis for the cost of the Impedance raster.  By normalizing these costs the tool provides the user more control over the shape of the final flow lines through the Source and Destination Weight sliders.4.png

 

To understand how cell size affects the total cost surface it helps to see a few examples.  In the following ArcScene screenshots we are using a Source location in the center of the US and the destination features are the centroids of a several countries in Africa. The CostDistance surface is shown floating over the “flat Earth” to provide a means of comparing the effects of the weight parameters and cell size. In each example the perspective and Z exaggeration are the same.

5.png

Source Attraction = 10 Destination Attraction = 1 Foreground Weight =8 Cell Size = 100,000 meters

6.png

Source Attraction = 10 Destination Attraction = 1 Foreground Weight =1 Cell Size = 100,000 meters

7.png

Source Attraction = 1 Destination Attraction = 1 Foreground Weight =10 Cell Size = 100,000 meters

8.png

Source Attraction = 1 Destination Attraction = 10 Foreground Weight =8 Cell Size = 100,000 meters

9.png

Source Attraction = 1 Destination Attraction = 10 Foreground Weight =8 Cell Size = 30,000 meters

 

 

In this last example, it was necessary to reduce the elevation exaggeration and to zoom out so the cost surface would display in a reasonable way.  The only difference between the two results is that the cell size has been reduced from 100,000 meters to just 30,000 meters.  The effect is that the number of “cost slices” is larger and so the CostDistance “elevation” is increased.

 

As previously stated, the new DFLT processes the Impassable features separate from Impedance features.  Impedance features are treated much the same as Impassable features in the first tool.  They are features the user would like the flow lines to avoid if possible but the flow lines will cross them if necessary to reach a destination location.  Impassable features represent a hard barrier in the cost surface and can be used to control the shape of the flow lines as in the example below.

10.png

Flow lines representing distribution of Coal by ship are crossing Central America regardless of Impedance weight

112.png

With the red barrier added flow lines now go around the tip of South America

 

 

As with the previous Flow Map tool, the output of this new tool will usually be the starting point to produce a final flow map with smoother more organic looking flow lines.  Once the lines are roughly where you want them the next step is to use Graduated Symbols to display the Distributed quantity.  The “GRID_CODE” field will contain values based on the input Distributed quantity field.  The values in this field will be equal to the Distributed quantity field for the smallest lines nearest to the destinations.  Each time a tributary flow line meets another line the combined line closer to the Source feature will have a GRID_CODE value equal to the sum of previous two tributaries. Temporarily placing labels on the lines and destination points makes this clear. In the examples above the lines are represented using 10 classes of line weights ranging from 1 – 10. Using rounded joints and ends on the lines also improves the output. After setting up graduated symbols many users may also prefer to run the Smooth Line tool on the flow lines. This will also add some curve to the flow lines which is often desirable. A good starting point is to use a tolerance value approximately four times the Cell Size used to create the flow lines. It is also recommended to select the PEAK algorithm and FIXED_CLOSED_ENDPOINT options. Even after these steps it may be necessary to make manual adjustments to the shape of the line.121.pngInstallation instructions:

 

After downloading and unzipping the Distributive Flow Lines tool you should see the following in the ArcMap catalog window.FLGCat.jpg

 

Expanding python toolbox will reveal one script tool as shown below:FLTcat2.jpg

 

By default, python toolboxes do not display the “pyt” file extension. To display this and other known extensions in ArcCatalog (or the catalog window in ArcMap) check the following:15.png

 

This dialog is accessible in ArcCatalog by clicking Customize > ArcCatalog Options.  Unchecking the highlighted option will show the following:FLGCat1.jpg

 

We hope you find the new tool useful and the tips above help get you started experimenting with the tool.  We look forward to you comments.

 

NOTE:  The Distributive Flow Lines tool was updated  Oct 22, 2013 to address a backwards compatibility issue.  If you are using ArcGIS 10.1 please download the latest version  here.

 

NOTE:  The Distributive Flow Lines tool was updated  Jan 10, 2014.  If you are using ArcGIS 10.1sp1 or 10.2 please download the latest version.Contributed by Bob Gerlt.

Filter Blog

By date: By tag: