Suzanne-Boden
Esri Regular Contributor
since ‎07-01-2014
41 Badges Earned
Expert Discoverer
View All Badges Earned
yesterday
475 Posts created
150 Kudos given
51 Solutions
530 Kudos received
cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

My Ideas

(0 Idea Submissions)

Latest Contributions by Suzanne-Boden

IDEA
-->
0
0
961
BLOG
0
0
633
BLOG
0
0
481
BLOG
Exploring Data

The first part of step 2 flows directly from the analysis question. Spend time upfront identifying all the data, including attributes, needed to answer the question. Don't get halfway through a project and discover you're missing a key dataset.

The point of exploring data is to understand what valid things you can do with it. You should:
  • Examine the metadata. Note the spatial resolution and accuracy, coordinate system, when the data was collected and by whom, data use constraints, and other important information.
  • Explore all the spatial datasets you plan to use in ArcMap. Do the layers align properly together? Are some layers more generalized than others? Do some layers have a larger (or smaller) extent than needed?
  • Explore the attribute table for each layer and note the number of records and the attributes. Sort fields and look at field statistics to understand the values. Note obvious data entry errors and inconsistencies.
Preparing Data

Preparing data means ensuring datasets can be validly analyzed together and reducing processing time as much as possible. Data preparation tasks often include projecting data, reducing the spatial extent to the area of interest, deleting unneeded attributes, creating new attributes, cleaning up attribute values, and more.

Get the most out of ArcGIS geoprocessing tools. Many can be run in batch mode and some have options you can use to combine data preparation tasks. For example, inside the Feature Class to Feature Class tool, you can create a SQL expression to import only features located in your area of interest, you can exclude source attributes you don't need, and you can define new attribute fields that you do need. With one tool, you accomplish three preparation steps.

You can also create a model to automate data preparation—just drag the geoprocessing tools you need into a model window and define their parameters. Creating a model is an excellent way to visualize and order tasks as you go.

The same data may be used for multiple analyses. It's good practice to make a copy of the original data before you make changes to prepare for a specific project. This way, the original data is preserved for subsequent analyses—and you have something to go back to just in case.Example: Analyze Piracy Incidents

Let's look at a simple example to illustrate the importance of exploring and preparing data. The analysis question has been framed as:
  • In the years 2009 through 2011, what types of vessels were the most frequent victims of piracy in and around the Gulf of Aden and Arabian Sea?

The U.S. National Geospatial-Intelligence Agency (NGA) distributes Anti-Shipping Activity Messages (ASAM) reports, which include locations and descriptions of hostile acts against ships and mariners worldwide. You can download ASAM data in shapefile format from the NGA website.Explore the Data

After downloading the ASAM shapefile, add it to ArcMap. For geographic context, it helps to add a basemap; in this case, the World Topograpic Map from ArcGIS Online works well.

The map graphic on the right shows the global extent of the ASAM data (symbolized with red dots). A quick exploration of the attribute table reveals 6,158 records; an attribute that stores a world subregion code; and an attribute that stores the date each incident occurred—the date range is 5/1/1978 through 1/9/2012.  ASAM_fullextent1.png

To efficiently answer the analysis question, you need to reduce the data to incidents that occurred between 1/1/2009 and 12/31/2011 in subregion 62 (the NGA website lists subregion codes). You also need to do some other preparation work to resolve issues with this data.Issue 1: The data has no spatial reference.

When the ASAM shapefile was added to ArcMap, a message that the data was missing a spatial reference displayed. For shapefiles, spatial reference information is stored in the projection (.PRJ) file and it's not unusual for the .PRJ file to be missing, as it is here. So how do you know which coordinate system to use?

A good place to start is the Layer Properties dialog box. In the Source tab, when you look at the extent coordinates, the number of digits to the left of the decimals tells you it's a geographic coordinate system. ASAM_coordinates1.png
  • With geographic coordinate systems, the Left and Right extent values will have one to three digits to the left of the decimal, while the Top and Bottom extent values will have one or two digits to the left of the decimal.

Since the data has a global extent, it is reasonable to assign the WGS 1984 geographic coordinate system. If your analysis involved precise measurements, you would want to assign a suitable projected coordinate system for the area of interest as well.Issue 2: The data is more extensive than you need.

The ASAM data has a larger spatial extent and a longer temporal extent than you need. SQL queries will resolve these issues.

For this analysis, the initial data preparation tasks are:

1. Create a file geodatabase to organize the project data.

2. Import the ASAM shapefile into the file geodatabase.
  • Import only features within subregion 62 that fall within the date range 1/1/2009-12/31/2011.

3. Assign the WGS 1984 coordinate system to the geodatabase feature class. DataPrepModel1-300x174.png

The model graphic on the right shows the geoprocessing workflow.

A SQL expression was defined in the Feature Class to Feature Class tool dialog so only the incidents meeting the analysis criteria will be imported.
After running the model, you have a new feature class with 643 features. Now that the data is a more manageable size, before moving on to step 3 of the analysis process, you need to verify that all 643 features represent piracy incidents that occurred during the analysis timeframe.

In ASAM data, two fields contain date information (Reference and DateOfOcc). DateOfOcc stores mm/dd/yyyy, while Reference includes the year followed by a unique ID number. The Reference field was used in the Feature Class to Feature Class dialog's SQL expression for convenience, but it means there's cleanup to be done now. Sorting the DateOfOcc field reveals that four records have 2009 in the Reference field (reflecting the year the report was submitted), but their DateOfOcc values tell you the incidents actually occurred during the last week of 2008. These records can be deleted. Now there are 639 records to analyze.Issue 3: Attribute values are inconsistent.

Next, you need to make sure all records are incidents of piracy. The Aggressor field stores this information. Sorting the Aggressor field reveals multiple values, the vast majority of which contain some variation of "pirate."  UniqueValuesLegend1.png
  • Tip: A quick visual method to understand how the Aggressor values vary is to open the Layer Properties dialog box and symbolize the layer by Unique Values based on the Aggressor field. Each unique aggressor value displays and the Count field tells you how many of each.

After exploring the Aggressor values, you can create an attribute query to select all records that have one of the "pirate" variations in the Aggressor field, then switch the selection to see how many records don't directly reference pirate aggressors. In this case, only 16 incidents have Aggressor values that don't reference pirates.

You need to explore the incident descriptions to determine whether any of these 16 likely were attacks by pirates. If they clearly did not involve pirate aggressors, delete the records. In this case, the incident descriptions don't clearly rule out pirate aggressors, so you choose to keep the records but categorize them to account for the ambiguity. You use the Field Calculator to change those 16 Aggressor values to Possible Pirates, and you clean up the remaining records so they all have an Aggressor value of Pirates.Issue 4: The table has duplicate records.

With the DateOfOcc field sorted, a look at the Reference field shows something alarming. There seem to be many duplicate records—records with the same Reference value and identical data in the other attribute fields. How could this be? Did something go wrong when you converted the shapefile to a geodatabase feature class?  DupValuesTable.png

This is the time to go back to the original data and determine if the duplicates existed there. Because of the number of features in the shapefile, it's efficient to summarize the Reference field, which outputs a table with a count of each unique reference value in the shapefile.

After creating the summary table and sorting the Count_Reference field, you see that many Reference numbers have a duplicate. To find out exactly how many, select the records whose Count_Reference value is equal to 2. It turns out that 493 Reference numbers have a duplicate in the source data you started with. Determining how many of those 493 are in your geodatabase feature class requires more work.

Just like you did for the shapefile, in the Incidents to Analyze layer, summarize the Reference field to output a table with a count of reference values, then select the records whose Count_Reference value is equal to 2. It turns out that 170 records have a duplicate. Is there a way to easily remove 170 duplicate records? Yes, fairly easily. Here's how:
  1. Join the summary table of reference values to the Incidents to Analyze table.
  2. Select records whose Count_Reference value is equal to 2—as expected, there are 340 selected records.
  3. Start an edit session, then use the Delete Identical tool to delete the selected records that have the same Reference value as another record (note that before running the Delete Identical tool, you must remove the table join—the selected records remain selected after the join is removed).

Final result: 469 incidents ready to be used as input in step 3 of the GIS analysis process.

All this data exploring and preparing really only required a couple of hours of focused work. Of course, the incident victim values likely need cleanup as well—but we'll end the example here as the boundary line between steps 2 and 3 of the GIS analysis process is not a solid fence.Takeaway: When preparing data for analysis, you have to make decisions and live with some uncertainty. Your specific analysis criteria and how well you know the topic determine how far you go to clean up the data. To avoid creating or propagating errors, carefully explore the data and document any data preparation you do and why you chose to do it. Remember that a model is a valuable tool to document your workflow and help others better understand your analysis results.

The data you use for a GIS analysis project may not be perfect and it may not fit your needs exactly. But planning and preparation go a long way toward making sure the data generates reliable results you can confidently share with others.

Want to learn more best practices for GIS analysis? Here are courses that can help. -->
0
0
3159
IDEA
-->
0
0
773
BLOG
0
0
967
IDEA
-->
0
0
569
IDEA
-->
0
0
1074
IDEA
-->
0
0
1074
BLOG
0
0
4603
BLOG


The steps below show a quick way to create a Python script using what I call the F-C-P-M-T method (that's Find, Copy, Paste, Modify, Test). Like developers who find sample code and modify it for their own purposes, you can find example Python script code in the ArcGIS Desktop Help and modify it for your needs. To work through the steps, download and unzip the data.

The steps assume you have ArcGIS 9.3 or 9.3.1, Python 2.5, and PythonWin installed on your computer or a network location you can access and write data to.
  • Note: If you are going to write Python scripts, you'll want to have PythonWin, an application that provides a friendly script-creation, testing, and debugging environment. PythonWin 2.5.1 is included on the ArcGIS 9.3 and 9.3.1 installation media, but it isn't installed by default. You can also get PythonWin version 2.5.4 from http://www.python.org/download/releases/2.5.4/.
The Scenario

Suppose you work in public safety for the city of San Diego and your task is to identify trolley stops that may need to be evacuated and closed in the event of an emergency on a nearby freeway. You have feature classes representing the San Diego freeways, trolley lines, and trolley stops. You want to write a script to buffer the freeways, then overlay the trolley stops and the buffers to output all the trolley stops that may be impacted in an emergency to a new feature class.

Why bother to write a script to do this fairly simple GIS operation?
  • Reusability—you can quickly change the buffer distance if you need to just by editing the script, then run the script to create new buffers.
  • You can share the script file with colleagues, who may want to use the script for their own purposes. They could modify the buffer distance and specify different feature classes to create data for their own projects, all without ever opening ArcMap or ArcCatalog.
  • You can also save the script as a custom tool and include it in a geoprocessing model.
Step 1: Find Geoprocessing Tool Script Code
  • Start ArcMap, open a new empty map document, and display the ArcToolbox window. Click to enlarge
  • Expand the Analysis Tools toolbox and the Proximity toolset.
  • Right-click the Buffer tool and choose Help.
  • In the help window, scroll down until you see the Scripting syntax section.
  • Below the syntax is a script example. Expand the example to see the code.
Step 2: Copy/Paste Example Code into a New Script
  • Click Start > All Programs > Python 2.5 > PythonWin.
  • From the File menu, choose New > Python Script and click OK.
  • With the Script1 window active, from the File menu, choose Save As and save the script with the name BufferIntersect.py in a local folder.Click to enlarge
  • Go back to the ArcGIS Desktop Help window, select all the code in the script example, right-click the selection, and choose Copy.
  • In PythonWin, right-click inside the BufferIntersect.py window and choose Paste.
  • Close the ArcGIS Desktop Help window and navigate to the Analysis Tools toolbox > Overlay toolset > Intersect tool.
  • Right-click the Intersect tool and choose Help.
  • Again, scroll down to the script example, select all the code, right-click the selection, and choose Copy.
  • In PythonWin, press Enter to move your cursor to a new line, then right-click and choose Paste.
  • Close the ArcGIS Desktop Help window and minimize ArcMap.

At this point, you have created a script that contains example code that will run the two geoprocessing tools you need for your project. Because you have combined the examples, your script contains some extra code you don't need, plus of course the example code refers to data you don't have. The first thing to do is examine the code and remove the duplicate lines.Step 3: Modify the Code to Meet Your Needs

Python scripts that incorporate ArcGIS geoprocessing functionality start with these two lines of code:import statement

In the script window, you see this code at the very top of the first example and again in the second example, which starts with a comment that explains the code's purpose. Including comments in a script is good practice so that others can understand what the code is doing. In Python, comments display in green italic text and are always preceded by a pound sign (#). When the script is run, comments are ignored.
  • Cut the Purpose and "Create the Geoprocessor object" comment lines and paste them at the top of the script.
  • Edit the purpose text to reflect this project's purpose and change the second comment text to: Import ArcGIS scripting module, create the Geoprocessor object, and specify the workspace and toolbox. Click to enlarge
  • Add a Date Created comment as the first line in the script.
  • In the gp.workspace line, change the path to reflect the path where you saved the San Diego geodatabase. Note that Python is case-sensitive.
  • The gp.toolbox = "analysis" line of code means that the geoprocessing tools referenced in the script are located in the Analysis toolbox. This is true for your project, so you don't need to change it.
  • Select and delete the second (duplicate) lines that import the ArcGIS scripting module and create the geoprocessor object.
Step 3a: Modify the Code for the Buffer Tool

The example code for the buffer tool includes all seven parameters accepted by the tool. Only the first three parameters are required (input feature class, output feature class, and buffer distance).
    • To learn more about the parameters, open the ArcGIS Desktop Help for the tool and look at the scripting syntax.

For this project, you only need the required parameters. Notice that parameters are separated by a comma in Python.
  • Edit the comment text about what is being buffered to reflect this project.
  • Change the input and output parameters as follows:
    • Input feature class: freeways (remember, Python is case-sensitive)
    • Output feature class: Freeway_Buffers

In the example code, the buffer distance is a string (surrounded by quotes). For this project, you are going to enter an explicit numeric distance. Click to enlarge
  • Select and delete "Distance" and replace it with 1320.
    • The distance is specified in feet because the freeway data units are feet.
  • Delete the last four optional parameters from the example code.
Step 3b: Modify the Code for the Intersect Tool

The example code for the intersect tool includes error handling. The try and except statements are used to handle unexpected errors during the code execution. For this simple project, you won't include error handling. Additionally, the example includes multiple geoprocessing operations and optional parameters for the intersect operation. For this example, you only need the intersect code (gp.Intersect_analysis) and you will use only required intersect parameters (the inputs and output).
  • Delete the try: statement and the except: statement.
  • Delete the comment about setting the workspace and the line of code that specifies the workspace—you have already taken care of this at the top of the script.
  • Edit the comment for the intersect code (starts with # Process) to describe the purpose of this intersect operation.
  • Use the Backspace or Delete key to move the comment and intersect code up to be just below and aligned with the buffer code (remove the indentation from the intersect section).
  • Modify the intersect code as follows:
    • Inputs: trolleystop and Freeway_Buffers
      • Notice that the input feature classes are separated by a semicolon and that quotes are used only before the first input and after the last input. Be sure to keep this formatting.
    • Output: StopsNearFreeways
  • Delete the last three optional parameters.
  • Delete the comments and lines of code that buffer, clip, and generate statistics and delete the print statement at the bottom.

The code for the geoprocessing operations now meets the project needs. When the script is run, you want confirmation that the code is executing properly. You can add print statements to display a message in the Interactive Window after each tool runs.Click to enlarge
  • Place your cursor at the end of the buffer line of code and press Enter.
  • Type print "The freeway buffers have been created."
  • Place your cursor below the intersect line of code and type print "The intersect was completed."
  • Save the script.

That's it, you're done. The script is ready to test.Step 4: Run the Script
  • Click the Run button Run, then click OK.
  • If you can't see the Interactive Window, from the Window menu, choose Interactive Window. (If you don't see it on the Window menu, use the View menu.)Click to enlarge

After a moment, in the Interactive Window, you should see the print statements you entered.
  • If your script does not run correctly, carefully compare your code to the code shown in the last graphic of step 3, make any fixes, save and run the script again. Note, however, that if the buffer portion ran correctly the first time but the intersect did not, and you re-run the script, you will get an error because the buffer output will already exist in the geodatabase. Solution? Use ArcCatalog to delete the buffer output (Freeways_Buffer), then run the script again.

If you like, you can verify the data was created using ArcMap.Click to enlarge


  • In ArcMap, click the Add Data button, navigate to the location where the data was output, and add the Freeway_Buffers and StopsNearFreeways feature classes.
  • Explore the data as desired. When you're finished, exit ArcMap and PythonWin.
Want to Learn More?

This was obviously a very simple scripting exercise intended to show how you can find example Python code and combine it to create a script that does something you need to do. To create robust scripts, however, you need to invest some time learning Python concepts and syntax.
  • The training options linked to above will get you well on your way to being a script master.
  • A couple of recommended books for beginners are Learning Python, 3rd Edition (by Mark Lutz) and Core Python Programming, 2nd Edition (by Wesley Chun). Many books have been written, however. One of them is sure to appeal to you.
-->
0
0
1135
BLOG
2
6
10110
BLOG
Step-by-Step Example

For this example, suppose you are researching factors that impact the water quality of Lake Okeechobee in Florida. Your study area is the counties that border the lake. You have a layer representing the lake and a layer of all Florida counties. You want to create a simple map that shows the study area, then export the map as a graphic to include in your report.
  • Add the data to a new map document.
  • Double-click the county layer name in the Table of Contents to open its Layer Properties dialog box.
  • In the Symbology tab's Show area on the left side of the dialog box, click Categories. Unique values is selected by default.
  • On the right choose the Value Field you want to symbolize on in the drop-down list. In this example, we choose the field that stores the county names.

In the symbol area, there is one default symbol with "all other values" displayed next to it.
  • Click the Add Values button.
  • In the Add Values dialog box, choose the counties that border the lake. You can hold down the Ctrl key to select multiple counties in the list. Click OK.

Back in the symbol area of the Layer Properties dialog box, the symbols for the counties you just added are all unique (a different color). You want them to have the same symbol.
  • Hold down your Shift key and click the top county name symbol (not the "all other values" symbol), then click the bottom symbol so that all the unique county symbols are selected. Right-click the selection and choose Properties for Selected Symbols.
  • In the Symbol Editor, set the desired symbol properties (color, size, etc.), then click OK. We chose a light red fill and a darker red outline with a width of 1.5.
  • Double-click the symbol for "all other values," use the Symbol Editor to set its properties, then click OK.
  • Click in the Label column for the "all other values" symbol and type in the text that you want to display next to that symbol in the Table of Contents. You can also edit or delete the default heading text that will display over the unique county symbols if you want.
Click to enlargeNote: Sometimes the outline of the symbols for the features of interest will be obscured by the symbols for the other features. To make sure the unique symbols are most prominent, you can set the symbol levels for the layer.
  • Below the symbol area in the Layer Properties dialog box, click Advanced > Symbol Levels.
  • In the dialog box, select the symbol you want to be least prominent and click the Move to Bottom arrow button to move it to the bottom of the list.
Click to enlarge
  • Click OK, then click OK again to close the Layer Properties dialog box and examine the map.
Click to enlarge

For this example, you would want to zoom into the study area counties and display their labels (use the Label Manager to create a label class that labels only the counties of interest—see our previous post about creating label classes).Click to enlarge

You could export the zoomed-in map to a graphic file (File menu > Export Map), then zoom back to the previous map extent and export the map showing the location of the study area within the state. Both map graphics could be inserted into your report to provide visual context.

That's it. Using basic functionality available through the Layer Properties dialog box, you can quickly create symbology to enhance a simple mapping project.

To assign a different symbol to one or a few features in a layer, you could also:
  • Make a copy of the layer and do a definition query on the copy to display only the features of interest.
  • In the original layer, select the features of interest and then create a selection layer of them.

These methods have their advantages. But for a quick job, the unique values approach works well. -->
0
0
711
BLOG
2
0
327
BLOG
  • Start ArcMap, open a new map document, and add the dataset you want to edit.
  • Display the Editor toolbar, start an edit session, then zoom into a feature you want to generalize.
  • Using the Edit tool, double-click the feature to display its sketch and update the edit task to Modify Feature.

We'll use the parcel shown in the graphic below as an example of how to complete the remaining steps. Notice that many vertices (green boxes) were created to represent the curved portion of the parcel's eastern boundary.Parcel sketch with many vertices
  • Click the Editor menu > More Editing Tools > Advanced Editing.
  • On the Advanced Editing toolbar that displays, click the Generalize tool.
Advanced Editing toolbar
  • In the dialog box, for Maximum Allowable Offset, enter 2.
    • Note: This value controls the degree to which the feature geometry will be simplified. For this example, an offset of 2 feet works well. When working with your own data, if you don't like the result after generalizing, you can click Undo, then repeat the preceding steps and enter a different offset value until you find an acceptable value.
  • Click OK.

You see the selected feature, which has been generalized. The curved shape of the example parcel's eastern boundary is still apparent.Selected parcel after generalizing
  • Double-click the feature to see its modified sketch.
Generalized parcel's sketch

Only six vertices are now used to represent the curve.
  • Save your edits and stop editing.

If you have multiple features to generalize, you can select them all at once and use the Generalize tool as instructed above. To create a generalized output feature class from an existing feature class, use one of the generalization tools found in the ArcToolbox Data Management Tools toolbox. -->
0
0
1847
Activity Feed