topic Easy way to remove collars from GeoTIFF of USGS Topo? in Imagery and Remote Sensing Questions

Easy way to remove collars from GeoTIFF of USGS Topo?

SeanWineland — Sat, 28 Oct 2017 21:18:55 GMT

I have downloaded a lot of GeoTiffs of historical topo maps from the USGS (https://ngmdb.usgs.gov/topoview) at the 1:24000 scale. I am wanting to Mosaic these together to eventually clip to watershed boundaries, however the white collars on the outside are proving to be a headache. I have not been able to find a solid answer on these forums or anywhere else on the internet of how exactly to remove the collars.

Does anyone have an answer?

Any help would be appreciated!

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

DanPatterson_Retired — Sat, 28 Oct 2017 22:51:02 GMT

page not found error

I don't imagine that it is as simple as clipping the collar. That would only be useful if the tiles matched at the edge when clipped. I suspect that there is some rotation and perhaps some overlap?

Perhaps you can provide some details as to whether there is overlap in the adjoining sheets and whether you can convert the 'collar' to nodata for use in a Con statement (details later)

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

XanderBakker — Sat, 28 Oct 2017 23:48:08 GMT

The "n" and "g" the the beginning of the URL are switched, it should be this: https://ngmdb.usgs.gov/topoview/

I just download 2 adjacent tiles and see what you mean there is overlap since the maps include all the legend and metadata too:

However, you could script a solution with Python. In the xml file of the data you will see information on the boundng box of the actual data:

Which corresponds to the map (looking at a corner):

So you could for each TIFF read the bounding box from the xml data and apply an Extract by Rectangle—Help | ArcGIS Desktop to get the part you are interested in.

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

XanderBakker — Sat, 28 Oct 2017 23:50:49 GMT

In additional, you would have to transform the decimal degrees from the bounding box to North_American_1927_Polyconic, since the tiles have projected coordinates. See the worldfile (tfw):

and the coordinate files (.prj)

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

SeanWineland — Sun, 29 Oct 2017 00:08:52 GMT

Thanks for your reply! I have never used python before, I have only ever done simple geoprocessing in ArcMap. Where do I find the .xml and .prj files and how would I do the transformation you are talking about?

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

SeanWineland — Sun, 29 Oct 2017 01:40:24 GMT

Okay, so I figured out how to open the files and actually successfully executed the extract by rectangle in Python. However, the output is just blank. So im thinking the bounding box coordinates are the extent of the whole image including the white collar? Not entirely sure. How would you successfully extract just the image?

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

XanderBakker — Sun, 29 Oct 2017 15:56:06 GMT

In order to extract the rectangle correctly you would have to specify the rectangle using the coordinates of the TIFF (the projected coordinates and not the geographic coordinates). To do this right you would have to project the decimal degrees found in the xml to the coordinate system "North_American_1927_Polyconic" (at least that is the coordinate system from the example tiles I downloaded). The challenge here might be the fact that this projected coordinate system is not a standard coordinate system recognized by ArcGIS as shown below:

Now it seems to be aligning with the basemap if I use the first transformation offered to me:

... and some changes through the years in the flow of the river become clear:

If I have some time, I will see what I can come up with to process the data. This may take some time though.

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

curtvprice — Sun, 29 Oct 2017 17:06:34 GMT

If you aren't picky about dates on the historical topos, this same information is available in the basemaps Esri provides, no collar issues. I would also look around local (state) websites, searching under the keyword "drg" for data stores of collar-clipped DRGs. It is extremely likely this has been done and the data is available for your area.

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

curtvprice — Sun, 29 Oct 2017 17:08:56 GMT

The Clip_management tool may be a better choice. Especially since the edge of the quad is not a rectangle - it's a polygon - which you would to generate and supply to the tool to clip to the edge of the map.

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

XanderBakker — Sat, 11 Dec 2021 11:02:02 GMT

Hi Sean Wineland ,

Just tried something out on a single TIFF, but I am a bit disappointed by the result, since a strip of white remains on the left side, probably since the coordinates may not have been transformed correctly:

Find below the code I used:

def main():
    import arcpy
    tiff_in = r'C:\GeoNet\TopoView\MA_Boston North_350031_1956_24000_geo.tif'

    tiff_out = r'C:\GeoNet\TopoView\tiff\MA350031.tif'
    fc_out = r'C:\GeoNet\TopoView\test.gdb\polygon_rectangle'
    sr_xml = arcpy.SpatialReference(4326)
    trans = "NAD_1927_To_WGS_1984_3"

    xml_file = tiff_in + ".xml"
    prj_file = tiff_in + ".prj"
    sr_tif = arcpy.SpatialReference(prj_file)
    polygon = getRectanglePolygon(xml_file, sr_xml, sr_tif, trans)

    arcpy.CopyFeatures_management([polygon], fc_out)

    rectangle_extent = "{0} {1} {2} {3}".format(polygon.extent.XMin, polygon.extent.YMin, polygon.extent.XMax, polygon.extent.YMax)

    arcpy.Clip_management(in_raster=tiff_in, rectangle=rectangle_extent,
                          out_raster=tiff_out, in_template_dataset=fc_out,
                          nodata_value="256", clipping_geometry="NONE",
                          maintain_clipping_extent="NO_MAINTAIN_EXTENT")


def getRectanglePolygon(xml_file, sr_in, sr_out, trans):
    dct_bounding = getBoundingCoords(xml_file)
    polygon = createPolygonFromCoords(dct_bounding, sr_in, sr_out, trans)
    return polygon

def getBoundingCoords(xml_file):
    lst_tags = ["westbc", "eastbc", "northbc", "southbc"]
    dct = {}
    with open(xml_file, 'r') as f:
        for line in f:
            line = line.replace("\n", "")
            for tag in lst_tags:
                if tag in line:
                    val = getValue(line, tag)
                    dct[tag] = val
    return dct

def createPolygonFromCoords(dct, sr_in, sr_out, trans):
    xmin = dct["westbc"]
    xmax = dct["eastbc"]
    ymin = dct["southbc"]
    ymax = dct["northbc"]
    lst_pnt_in = [arcpy.Point(xmin, ymin), arcpy.Point(xmin, ymax),
                arcpy.Point(xmax, ymax), arcpy.Point(xmax, ymin),
                arcpy.Point(xmin, ymin)]
    lst_pntg_in = [arcpy.PointGeometry(pnt, sr_in) for pnt in lst_pnt_in]
    lst_pntg_out = [pntg.projectAs(sr_out, trans) for pntg in lst_pntg_in]
    lst_pnt_out = [pntg.firstPoint for pntg in lst_pntg_out]
    return arcpy.Polygon(arcpy.Array(lst_pnt_out), sr_out)

def getValue(line, tag):
    start_tag = "<{0}>".format(tag)
    end_tag = "</{0}>".format(tag)
    pos1 = line.find(start_tag)
    pos2 = line.find(end_tag)
    txt = line[pos1 + len(start_tag):pos2]
    return float(txt)

if __name__ == '__main__':
    main()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This could be place in a loop to run through the TIFF files you have, but first the possible shift in coordinates should be examined and corrected.

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

SeanWineland — Mon, 30 Oct 2017 02:22:31 GMT

Curtis,

Thank you so much for this information, I found DRG (.sid) files for the whole state I am working with. Any idea of how to mosaic these together? I will post another thread in this forum asking the same question.

Thank you all so much for your help! xander_bakker‌ as well even though I might not be using the Python Script you came up with I appreciate all your help as well

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

curtvprice — Mon, 30 Oct 2017 02:32:12 GMT

I would suggest not mosaicking them together (Mosaic To New Raster could do this in theory) but instead add them to a referenced Mosaic dataset so you can keep them in their zippy tiny compressed .sid format and yet display them seamlessly. Much more efficient way to deal with a large collection of rasters like that. Once they are loaded to a Mosaic Dataset you can clip out what you want (if you must) with the Clip_management tool.

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

DanPatterson_Retired — Mon, 30 Oct 2017 02:46:32 GMT

You can't mosaic sid files together in ArcMap... if memory serves you need FME

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

curtvprice — Mon, 30 Oct 2017 02:48:05 GMT

Wow that would be a pain, so I guess you'd have to Copy Raster them to another format and then mosaic them. Another argument to leave them as they are in .sid and use a mosaic dataset to display them together on the fly.

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

CodyBenkelman — Mon, 30 Oct 2017 06:58:39 GMT

Sean

have you looked at the historical USGS topo maps available as free services through ArcGIS Online? (ArcGIS Online subscription required)

http://www.arcgis.com/home/item.html?id=40109482569c4bcc873dfa7fec7c167e

There are tens of thousands of maps mosaicked and ready to use, at different scales and from different dates...

http://www.arcgis.com/home/item.html?id=40109482569c4bcc873dfa7fec7c167e

http://www.arcgis.com/home/item.html?id=be337c042cd4476e9ba29d4c0e26b5fb

many more

If you really need to create your own mosaic, I'd use a Mosaic Dataset, and define the seamline for each map using the extents of the map area (inside the "collar"), then mosaic method = seamlines will hide all the collars. This will be a dynamic mosaic, allowing you to expose the collar of any selected map if desired.

If you really need to output a new file, my first question is "Are you sure?", but you can export or cache the dynamic mosaic to persist a static image.

http://esriurl.com/ImageManagement --> look for the Scanned Maps workflow.

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

SeanWineland — Tue, 31 Oct 2017 19:10:59 GMT

Cody, I tried the link to the historical maps database but it did not work, i tried to sign in and it told me that I was not a member of this organization. Is it possible to run raster analysis on these basemaps? (supervised classification?)

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

CodyBenkelman — Tue, 31 Oct 2017 19:20:30 GMT

Sean

My apologies. When I inserted those links, I edited the visible text but didn't realize the hyperlinks were still going to an internal AGOL organization. Please try them again - they should be accessible to anyone with an ArcGIS Online account.

No, maps are not suitable input to classification. Image classification is intended for imagery, not the thematic representation you'd find in a map. I've never heard of anyone attempting it, but I'm confident you won't get results that make any sense.

Separate from the issue of mosaicking maps and removing the collars, or accessing image services, perhaps we can help you better if you explain your objectives, and ideas on how you're planning to approach the problem.

Cody B.

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

SeanWineland — Fri, 03 Nov 2017 13:23:46 GMT

I see, I do not have an arcgis online subscription anyways.

My objectives are to somehow calculate the area of historical strip mining within individual watershed boundaries (I have shapefiles of watershed boundaries, and historical topo map files (DRG's). But this is proving to be quite difficult in every aspect. The DRG files I have downloaded are (.sid) files so I had to transform them into .tiff files to work with them. I'm currently trying to mosaic them together but this process has been running for 4 days now and I'm just really not sure if what I'm doing will work out.

My professor thinks I can clip the mosiac to each watershed boundary and then run a supervised classification on the clipped image to extract areas of strip mining and non-strip mining, then perform a raster to vector on that image to calculate the area of strip mining.

Any advice on how to achieve this goal would be greatly appreciated.

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

DanPatterson_Retired — Fri, 03 Nov 2017 13:37:00 GMT

Seems like overkill, unless you are doing dozens of years. you have ruled out the obvious solution of digitizing the areas manually? Human-ware would easily skip the collars and remove the 'errers' associated with doing a classification. The eye is better than the machine in many cases and it least it would get you going forward and you can fill in the fuzzy process gaps later

Re: Easy way to remove collars from GeoTIFF of USGS Topo?

SeanWineland — Fri, 03 Nov 2017 15:33:47 GMT

I have not ruled it out, was just hoping to automate the process given that I have 67 individual watersheds I need to do this for. I guess I can go with digitizing to get started. If I union the digitized polygons with the watershed boundary would this allow me to calculate the percentage of area within the watershed covered by mining?