View GeoPDF Files in ArcMap

[DouglasCaldwell]

GeoPDF formatted data represents a large and growing source of information. The US Geological Survey is creating all new US Topo products in GeoPDF format and the National Geospatial Intelligence Agency is releasing scanned map products in GeoPDF format.

It would be a great help if ArcMap would be able to display these GeoPDF products ... at least as a background map ... if not exploit their content further.

[MaxSchneider]

Seriously, what is taking ESRI so long for to accept and make this format usable ...Probably trying to figure out how to charge us another $2500.00 for the extension... Why ESRI??? Haven't we given enough?

[JohnNiles]

GeoPDFs are completely useless without the ability to open in ArcMap... get on it already!

[ErinFashoway]

This would be a great addition to ArcMap functionality!

[JohnNiles]

Really ESRI? This was posted three years ago.. Since the USGS made the huge mistake of producing all the topos in GeoPDF format, there has been no effort by anyone to make these products actually usable for the GIS user. It's a matter of being able to open a georeferenced file open in ArcMap - one of the core uses of the program!! That's one of the things it does!! Hell, doesn't ESRI have a contract with the DOI to begin with?? This whole poderous failure is just astounding. 

[DaveSivertsen]

Would like to bring in USGS maps.
Would like to generate maps for generic Adobe Reader users.

[DustinWittwer1]

TerraGo is a good workable solution but it is SO OVERPRICED!  It would be nice if it were a natively supported format.

[jimschmidt2]

Here's one way to convert Historic GeoPDF's in polyconic projections to GeoTiffs geo-referenced to lat-longs in ArcMap using GDAL.

Part1
 

1. Download the historic maps desired in geo-pdf format using the National Map Viewer.

 

1. Unzip the files.

 

1. Remove any spaces in the pdf filenames.

 

1. Install a software package that includes GDAL (Geospatial Data Abstraction Library). I installed MapServer for Windows (MS4W) version 3.0.6 (http://www.maptools.org/ms4w/index.phtml?page=downloads.html).

 

1. Create a batch file to use the GDAL translate utility to convert the PDF’s to GeoTiffs.

(Example: gtiffBatch.bat)
 
for /R C:\USGS_Topos1 %%f IN (*.pdf) do gdal_translate.exe  -of GTiff %%f C:\Gtiff\%%~nf.tif
 
where C:\USGS_Topos1 is the directory where I have the pdf files and C:\Gtiff is the output directory where the GeoTiff files will be written. %%f is the filename wildcard.
 

1. Open the MS4W shell script window (should be an icon on your desktop once installed). Type gtiffBatch.bat  (see above) to execute the batch file.  It will process all the pdf files in the source directory and write out the GeoTiff files to the destination directory. It takes about 10 seconds per file to do the conversion.

Note: If you are not dealing with Historic maps in the polyconic projection, you can stop here. The GeoPdfs should be useable in ArcMap as is. ArcGIS does not recognize the polyconic projection, so you need to do the following steps to convert the GeoTiffs to a lat-long georeferencing scheme.
 
 
 

[jimschmidt2]

Converting Historic GeoPDF's in polyconic projections to GeoTiffs geo-referenced to lat-longs in ArcMap using GDAL:

Part II -

1. Create a batch file to use gdalinfo to read the header information for each GeoTiff and write it out to a text file. Run  the batch file from the MS4W shell prompt.

 
Example ginfoBatch.bat contents:
 
for /R C:\Gtiff %%f IN (*.tif) do gdalinfo.exe %%f >> C:\Gtiff\%%~nf.txt
 
where C:\Gtiff is the directory where the GeoTiff files are located and also where the text files will be written.

1. Run a python script to extract the control point information from the files created in step 7, convert lat-longs coordinates to decimal degrees, and output the data to text files that can be loaded as Control Point tables with the Georeferencing toolbar in ArcGIS.

 
Sample Script (See Appendix): ControlPoints.py – Place the script on your desktop. Go to Start—Run—cmd to start the DOS command shell.  Change directories to your desktop (type cd desktop). Execute the python script by typing:
 
C:\python26\Arcgis10.0\python Controlpoints.py
 
Note: You may need to change the command to reflect your current version of python.
 
Output files with control point information will be re-named to start with CP instead of CA. If you are working with files from a state other than California, edit the ControlPoints.py script to rename as desired.
 
Note also that the ControlPoints.py script expects the control point information in the Gdalinfo output files to be in a fixed format. If that format varies, the script may not extract the correct control point data.
 

1. To change the georeferencing of the geotiff files created in Step 5 from Polyconic to geographic, load each geotiff file into ArcGIS. Set the Data Frame coordinate system to Geographic – NAD27.

 

1. Load the Georeferencing Toolbar. Make sure the Layer shown on the toolbar is the layer you want to georeference. Click on the View Link Table button. In the Table window, click on the Load button and select the proper Control Point file (starting with “CP”) for the geotiff file shown on the georeferencing toolbar (The Control Point files were created in Step 8). Ignore any warning messages about collinear points.  Click OK.

 
 

1. On the Georeferencing Toolbar dropdown menu, select Update Georeferencing.

Note: If Update Georeferencing is greyed out, try Update Display first.
 

[jimschmidt2]

Converting Historic GeoPDFs in polyconic projections to GeoTiffs geo-referenced to lat-longs in ArcMap using GDAL:



Part III

Appendix - Sample python script for creating ArcGIS compatible Control Point tables:

ControlPoints.py:
 
 
import glob
from decimal import Decimal
list_of_files = glob.glob('C:\Gtiffinfo1\*.txt')# create the list of file
for file_name in list_of_files:
  FI = open(file_name, 'r')
  print FI
  FO = open(file_name.replace('CA_', 'CP_'), 'w')
  buffer = []
  blnk = ' '
  newline = '\n'
  for line in FI:
    if line.startswith("Up"): #Read Control Point info
    x = line[15:25]
    y = line[26:38]
    lond = line[41:44]
    lonm = line[45:47]
    lons = line[48:53]
    latd = line[57:59]
    latm = line[60:62]
    lats = line[63:68]
    londd = Decimal(lond.strip(' "')) # Convert to decimal
    lonmd = Decimal(lonm.strip(' "'))
    lonsd = Decimal(lons.strip(' "'))
    londecdeg = (londd + lonmd/60 + lonsd/3600)* -1 #Convert to decimal degrees
    londdstr = "%11.6f" % londecdeg
    latdd = Decimal(latd.strip(' "'))
    latmd = Decimal(latm.strip(' "'))
    latsd = Decimal(lats.strip(' "'))
    latdecdeg = latdd + latmd/60 + latsd/3600
    latddstr = "%9.6f" % latdecdeg
    buffer.append(x)
    buffer.append(blnk)
    buffer.append(y)
    buffer.append(blnk)
    buffer.append(londdstr)
    buffer.append(blnk)
    buffer.append(latddstr)
    buffer.append(newline)
    FO.write("".join(buffer))
  #now reset our state
    buffer = []

[jimschmidt2]

Python Script continued:

Part IV:

 elif line.startswith("Lo"):
    x = line[15:25]
    y = line[26:38]
    lond = line[41:44]
    lonm = line[45:47]
    lons = line[48:53]
    latd = line[57:59]
    latm = line[60:62]
    lats = line[63:68]
    londd = Decimal(lond.strip(' "'))
    lonmd = Decimal(lonm.strip(' "'))
    lonsd = Decimal(lons.strip(' "'))
    londecdeg = (londd + lonmd/60 + lonsd/3600)* -1
    londdstr = "%11.6f" % londecdeg
    latdd = Decimal(latd.strip(' "'))
    latmd = Decimal(latm.strip(' "'))
    latsd = Decimal(lats.strip(' "'))
    latdecdeg = (latdd + latmd/60 + latsd/3600)* 1
    latddstr = "%9.6f" % latdecdeg
    buffer.append(x)
    buffer.append(blnk)
    buffer.append(y)
    buffer.append(blnk)
    buffer.append(londdstr)
    buffer.append(blnk)
    buffer.append(latddstr)
    buffer.append(newline)
    FO.write("".join(buffer))
#now reset our state
    buffer = []
FI.close()
FO.close()

Typical output control point file  for one Historic Geotiff file looks like this:

-25894.864  4211779.501 -120.044975 38.040058
-25894.864  4144998.667 -120.042597 37.438378
 26704.302  4211779.501 -119.445806 38.040033
 26704.302  4144998.667 -119.448256 37.438356