topic Re: Creating a shovel test grid for Cultural Resource Management in Archaeology GIS Questions

Creating a shovel test grid for Cultural Resource Management

EricMai — Sat, 05 Aug 2017 16:39:01 GMT

I can't find a good amount of information on creating a grid within a set boundary and using the intersection points as points for data. I work in the archaeology field and we dig shovel tests on a 50 foot grid. I am trying to figure out a way of creating a grid before entering the field, where each intersection point already has a distinct GPS point, and data can be entered via a database app in reference to each one of these points. Can anyone help me through this process?

I understand the fishnet tool, but not sure how to use the intersections and create points for each, and additional attributes to be filled in later. And how could I label each point based off of that grid system of x, y? For example, transect 1, shovel test 1.

Re: Creating a shovel test grid for Cultural Resource Management

BrianWilson — Sat, 05 Aug 2017 18:08:42 GMT

I am not an archeologist, help me out here, when you say 50' grid I am assuming that's a polygon and you subdivide it into small squares? Or that's the size of one square in a big grid?

When you say you have GPS points, what do they look like? That is, GPS data wanders around, do you mean you have decimeter or better accuracy for a grid of points?

Writing a bit of code to generate a grid is the easy part for me. 🙂 Knowing what you want is the hard part.

If you have a picture or reference to a web site that would teach me more please post it and I will help you.

Re: Creating a shovel test grid for Cultural Resource Management

DanPatterson_Retired — Sat, 05 Aug 2017 18:49:13 GMT

You could either use the centroids of the fishnet to serve as your reference points... or convert the fishnet to points which will have a point at each intersection... you will then have to clean up the point file to remove duplicates.

All are doable, but I will spare you the tool links for now.

Once the geometry is created, then you will have to add fields to the table and calculate the long/lat values (easy) to tie them into your points. Add further text or numeric fields getting them in the order you want... don't skip the thought/workflow process when doing this or you will be jumping around the table.

Write it out on paper... why does it have to be an intersection of two lines? why not the centroid of a square?

Thought before tools and tools way before coding

Re: Creating a shovel test grid for Cultural Resource Management

EricMai — Mon, 07 Aug 2017 12:50:58 GMT

https://drive.google.com/open?id=0B9EAL0oxgEMwXzdFN3h4c1I0azA

I attached a link to a sketch of what I'm looking to do. Project areas are often not perfect rectangular grids so sometimes the grid is somewhat amorphous. An X axis is the Transect # (1,2,3,4...) and the Y axis is Shovel Test # (1,2,3,4...). From one 1,2 to 1,1, the distance is 50 feet, so yes, one block is 50' square.

So at each intersection, we are digging a shovel test pit and recording information. Mainly measurements, soil type, and artifact presence, if any. I want to have a GPS coordinate using WGS84 for each point so field surveyors can walk straight to that point without having to manually sight with compasses. So the GPS is generated in ArcMap before even heading out into the field.

I hope that is a little bit clearer.

Thanks Brian.

Re: Creating a shovel test grid for Cultural Resource Management

EricMai — Mon, 07 Aug 2017 12:53:16 GMT

I guess it could be a centroid. You're right. It achieves the same distance between points. It's just how I've always hand drawn it. What duplicates would there be though?

Re: Creating a shovel test grid for Cultural Resource Management

DanPatterson_Retired — Mon, 07 Aug 2017 13:08:00 GMT

Picture an intersection point of 4 square polygons... the intersection point will be contained in each polygon, hence, if you convert the polygon to points, you will have at least 4 identical points (even more if the intersection point is the start/end point of a polygon)

In short... centroids are best... just shift in your mind your lines so that they are offset by 1/2 half the distance between the points... hope you catch my drift

Re: Creating a shovel test grid for Cultural Resource Management

BrianWilson — Tue, 08 Aug 2017 20:18:25 GMT

Eric,

A friend told me building bicycle wheels was like knitting for him. For me it's coding. Sometimes I write code because it's fun and satisfying and so the "Thought before tools and tools way before coding" does not always apply for me. Also sometimes having EXACTLY the write tool is so satisfying that I just have to build my own.

I will look at your picture and notes again after lunch!

Brian

Re: Creating a shovel test grid for Cultural Resource Management

ChrisDonohue__GISP — Tue, 08 Aug 2017 21:20:59 GMT

As suggested by others, this can be done with the Create Fishnet (Data Management) tool

Create Fishnet—Data Management toolbox | ArcGIS Desktop

Note that checking off the 'Create Label Points (optional) checkbox in the tool will provide a points layer of the centroids. After that, it is just a question of culling down the points to just within your site and then adding whatever fields are needed.

Chris Donohue, GISP

Re: Creating a shovel test grid for Cultural Resource Management

BrianWilson — Wed, 09 Aug 2017 01:04:31 GMT

Further thoughts

Would this work as a suggested work flow?

Working in a map with local coordinate system in FEET,

1 Create a line to define the origin (1,1) and direction of the Y-axis.

2 Select a template polygon feature class to define extent.

3 Run tool to generate grid.

4 If you need to tweak the grid, do that now using ArcGIS edit tools.

5 Run tool to convert grid to points; the tool will also label

points (X,Y) starting at (1,1) by default.

6 If you need to tweak any POINTS you can do that now; move, add or delete

points and edit the labels in attribute data. The labels will be exported

to the GPX file so you can see them on the GPS screen.

7 Use tool to reproject to WGS84 and export to GPX file for field use.

(When I say "tool" here I mean some operation that might be a piece

of python or a model or a generic ESRI tool

for the purposes of discussion assume it exists.)

In step 1 I assume you draw a line starting at point (1,1) and going

up in the direction of "Y"

and that direction is not "North" Then the tool in step 3 works

"up" and to the right generating 50' squares

until it hits the edge of the template. It will generate complete

squares only, corners OUTSIDE the template

will not generate points.

My assumptions so far:

The template polygon feature class will have just one polygon in it
Each intersection is the location of shovel test. Using centroids is

a great way to use Fishnet but will create confusion when visualizing

the fishnet grid. The grid-to-point tool (step 5) will deal with corners.

Your GPS unit can accept a GPX file. Does it?

Re: Creating a shovel test grid for Cultural Resource Management

BrianWilson — Wed, 09 Aug 2017 19:38:25 GMT

I now have my "shovel test" tool generating a grid of polygons and points, it measures the extent of the template polygon to figure out the grid size and then clips points generated to be to the right and above the origin and inside the template.

Each point feature contains a "label" field that has the grid numbers as shown.

If you wanted tiles to start lower down you'd need to drag the baseline south and run the tool again.

Those are 50' tiles.

I can publish the code if you want it. I did this as an exercise. Drop me a line or reply here.

Re: Creating a shovel test grid for Cultural Resource Management

EricMai — Wed, 09 Aug 2017 21:54:26 GMT

Brian, this is exactly what I am looking for! I would love to see the code you used. Did you create a workflow model for the tool? Thank you thank you for this.

Re: Creating a shovel test grid for Cultural Resource Management

DanPatterson_Retired — Wed, 09 Aug 2017 23:17:18 GMT

There are code samples here

https://community.esri.com/blogs/dan_patterson/2016/09/09/numpy-snippets-3-phishnyet-creating-sampling-grids-using-numpy

here

https://community.esri.com/blogs/dan_patterson/2016/09/09/n-gons-regular-polygonal-shape-generation

and xander_bakker‌ posted some as well which also does the labelling, but I can't put my hands on it.

The first link enables, rectangles or hexagon sampling frameworks with or without rotation.

Re: Creating a shovel test grid for Cultural Resource Management

BrianWilson — Wed, 09 Aug 2017 23:59:33 GMT

Numpy is elegant, I suppose I will have to start over now. 🙂 Thanks for the link.

Re: Creating a shovel test grid for Cultural Resource Management

XanderBakker — Thu, 10 Aug 2017 01:00:27 GMT

I think Dan Patterson is referring to How do I create and display the row and column of a hexagon grid and How can I assign grid values to the squares of a fishnet grid? but both threads are more about the labeling.

Dan also had this example Grid Reference Numbers in Fishnet that was mentioned in the same thread here: https://community.esri.com/message/698234-re-how-to-convert-overlay-grid-to-features

Re: Creating a shovel test grid for Cultural Resource Management

BrianWilson — Thu, 10 Aug 2017 01:18:10 GMT

I set up a repository on github. I built a python toolbox too and I ran it successfully in ArcGIS Desktop 10.5.1

GitHub - Geo-CEG/shoveltest: Create grids to facilitate shovel tests for archaeology site work.

If you don't know how to use Github, please ask, I have been thinking of writing a blog post on it.

Please try this tool, I want it to work for you. I am sure there is plenty of tuning I could do on it but I'd like to hear if it is useful first. Next step in the workflow would be to convert the point grid into a GPX file; we should be able to do that in a model though.

In this screenshot I was able to edit the baseline then run the tool right in ArcMap to generate a new grid. Note there are a couple wandering isolated points in this picture -- if a complete square does not fit it will get dropped but the points that still fit inside the polygon were not dropped.

Re: Creating a shovel test grid for Cultural Resource Management

DanPatterson_Retired — Thu, 10 Aug 2017 01:29:46 GMT

Nice Brian!

GitHub's documentation is fairly good. But Dr. Google 'setting up a package' and other terms as well.

Also, you can link to ArcGIS Code Sharing (bottom left) and share your code from both, and maintain within GitHub

Re: Creating a shovel test grid for Cultural Resource Management

EricMai — Sun, 13 Aug 2017 14:12:32 GMT

Hi Brian.

Yes, I'm unfamiliar with GitHub. Can you explain how I would load these tools to arcmap? I've added the toolbox but there is some kind of error when I run the tool.

Executing: ShovelTest_Tool "C:\Users\ericv\Documents\ArcGIS\Web Maps\Mechanicsville, Virginia\Features.gdb\ShovelTestBaseline" "C:\Users\ericv\Documents\ArcGIS\Web Maps\Mechanicsville, Virginia\Features.gdb\ShovelTestArea" "C:\Users\ericv\Documents\ArcGIS\Web Maps\Mechanicsville, Virginia\Features.gdb" grid_poly grid_point 50 50
Start Time: Sun Aug 13 10:12:13 2017
Running script ShovelTest_Tool...
Parameter: baseline = C:\Users\ericv\Documents\ArcGIS\Web Maps\Mechanicsville, Virginia\Features.gdb\ShovelTestBaseline
Parameter: template = C:\Users\ericv\Documents\ArcGIS\Web Maps\Mechanicsville, Virginia\Features.gdb\ShovelTestArea
Parameter: output_workspace = C:\Users\ericv\Documents\ArcGIS\Web Maps\Mechanicsville, Virginia\Features.gdb
Parameter: output_polygon_features = grid_poly
Parameter: output_point_features = grid_point
Parameter: grid_cell_width = 50
Parameter: grid_cell_height = 50
baseline origin: -8609652,4521015 end: -8609420,4521206
Delete failed on "C:\Users\ericv\Documents\ArcGIS\Web Maps\Mechanicsville, Virginia\Features.gdb\grid_point". ERROR 000464: Cannot get exclusive schema lock. Either being edited or in use by another application.
Failed to execute (Delete).

Delete failed on "C:\Users\ericv\Documents\ArcGIS\Web Maps\Mechanicsville, Virginia\Features.gdb\grid_poly". ERROR 000464: Cannot get exclusive schema lock. Either being edited or in use by another application.
Failed to execute (Delete).

Grid size will be (4 x 7)

Traceback (most recent call last):
File "C:\Users\ericv\Documents\ArcGIS\Web Maps\Mechanicsville, Virginia\shoveltest-master\shoveltesttool.py", line 127, in execute
makegrid(baseline_fc, template_fc, workspace, output_poly, output_point, width, height)
File "C:\Users\ericv\Documents\ArcGIS\Web Maps\Mechanicsville, Virginia\shoveltest-master\shoveltest.py", line 153, in makegrid
cursor_pt.insertRow([p1, label])
RuntimeError: Cannot find field 'label'

Failed to execute (ShovelTest_Tool).
Failed at Sun Aug 13 10:12:13 2017 (Elapsed Time: 0.41 seconds)

Re: Creating a shovel test grid for Cultural Resource Management

XanderBakker — Sun, 13 Aug 2017 14:33:16 GMT

This is what can happen when the result from a previous run are still in the TOC of ArcMap. ArcMap will probably create an exclusive lock on those featureclasses and fail to delete them. So, remove

C:\Users\ericv\Documents\ArcGIS\Web Maps\Mechanicsville, Virginia\Features.gdb\grid_point

and

C:\Users\ericv\Documents\ArcGIS\Web Maps\Mechanicsville, Virginia\Features.gdb\grid_poly

...and try again.

Another question:

Do you have a folder called "Mechanicsville, Virginia" with a comma and a space"? If so, it is better to avoid using large names, spaces and special characters in folder names.

Re: Creating a shovel test grid for Cultural Resource Management

BrianWilson — Mon, 14 Aug 2017 21:40:26 GMT

If you search this forum for "schema lock" you will find hundreds of messages.

In my experiences, as with many things in ArcMap and ArcCatalog, quitting and restarting is the most reliable (and time consuming) way to deal with schema locks. There does not seem to be any consistent pattern to it that I have ever seen.

I find that if I try to keep ArcMap open and to run the tools in ArcCatalog it's worst - I have to close ArcMap to clear the lock before I can work in ArcCatalog. If the feature class or sometimes just the geodatabase is open in ArcMap it can get locked. I would think that it would only create locks when I start editing but that's not always true.

I have the best luck when working exclusively in ArcMap but even that's often problematic.

Re: Creating a shovel test grid for Cultural Resource Management

BrianWilson — Mon, 14 Aug 2017 21:41:44 GMT

I will try to write up something on github for GIS people soon.