Raster to polygon script loop failing!! error 99999!

[AliceDuff1]

Hello!

I am have already posted this in the general forum but i managed to solve that issue and now i have a new one!

I keep getting an error 99999 error message at line 37!
( gp.RasterToPolygon_conversion(INPUT_RASTER, Output_polygon_features, "SIMPLIFY", "VALUE") )
I am trying to make a script which selects every .png file in a folder beginning with the letters "LG". I then want the scipt create a shapefile, replacing the "LG" with "SH", and then i want the script to buffer that shapefile and rename the buffer with the first 2 letters being "SB"!

Can anyone see why this isnt working? I am very, very new to this and have been staring at this script pulling out my hair for days!! 😮

Here is the script:

# ---------------------------------------------------------------------------
# RASTER2POLYGON.py
# Created on: Wed Dec 01 2010 12:20:48 PM
#   (generated by ArcGIS/ModelBuilder)
# Usage: RASTER2POLYGON <INPUT_RASTER> <Output_polygon_features> <Raster_Buffer_shp>
# ---------------------------------------------------------------------------

# Import system modules
import sys, string, os, arcgisscripting

# Create the Geoprocessor object
gp = arcgisscripting.create()

# Load required toolboxes...
gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Conversion Tools.tbx")
gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Analysis Tools.tbx")

# Script arguments...

folder = "D:\\J04-0083\\IMAGEFILES"
for root, dirs, filenames in os.walk(folder): # returms root, dirs, and files
    for filename in filenames:
        filename_split = os.path.splitext(filename) # filename and extensionname (extension in [1])
        filename_zero = filename_split[0]

        try:
            first_2_letters = filename_zero[0] + filename_zero[1]
        except:
            first_2_letters = "XX"

        if first_2_letters == "LG":
            Output_polygon_features = "D:\\J04-0083\\ShapeFiles.gdb\\" + "SH_" + filename + ".shp"

            # Process: Raster to Polygon...

            INPUT_RASTER = os.path.join(root + "\\" + filename_zero + ".png")
            gp.RasterToPolygon_conversion(INPUT_RASTER, Output_polygon_features, "SIMPLIFY", "VALUE") 

            Distance__value_or_field_ = "5 Meters"
            Raster_Buffer_shp = "SB_" + filename + ".shp"

            # Process: Buffer...
            gp.Buffer_analysis(Output_polygon_features, Raster_Buffer_shp, Distance__value_or_field_, "FULL", "ROUND", "NONE", "")

[V_StuartFoote]

Alice,

I read the errors and looked at the listing of the IMAGES folder, and can see there are no duplicate file names and no file types other than .PNG.   Although, you may have some issue with splitting of the "LG_landelijk+hoogspanningsnet_Tennet+TSO+B.V._0000579962_10G179920.png" named image file.

The only question that still comes to mind... are you working with an open ArcCatalog session while debugging, and are you backing out of the ShapFiles.gdb folder and deleting or renaming the geodatabase.  I.e., starting fresh before attempting an additional run through the python script? If not, you should or you'll get errors similar to these.

So, do that. and also insert these print statements
[INDENT]
print(INPUT_RASTER)
print(Output_polygon_features)
print(Raster_Buffer_shp)
[/INDENT]
where they are declared in the script. They'll allow you to follow progress in the console as the script executes.

Stuart

[AliceDuff1]

Brilliant news! The script works now, the problem was that i was viewing the files in arc catalog, and outside of arccatalog there were other .aux and .xml files which were causing the error i think!!

Here is the new improved script

#--------------------- IMPORT --------------------------------------------------
import sys, string, os, arcgisscripting
gp = arcgisscripting.create(9.3)
# Load required toolboxes...
##gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Conversion Tools.tbx")
##gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Analysis Tools.tbx")

#--------------------- INPUT ---------------------------------------------------
gp.workspace = "D:/J040083"
gp.CreateFileGDB ("D:/J040083", "ShapeFiles.gdb")
folder = "D:/J040083/IMAGEFILES"

#--------------------- SCRIPT --------------------------------------------------
gp.OverWriteOutput = 1 # overwrite existing files (1 = True, 0 = False)

for root, dirs, filenames in os.walk(folder): # returms root, dirs, and files
    for filename in filenames:
        filename_split = os.path.splitext(filename) # filename and extensionname (extension in [1])
        filename_zero = filename_split[0]
        extension = str.upper(filename_split[1])

        try:
            first_2_letters = str.upper(filename_zero[0] + filename_zero[1])
        except:
            first_2_letters = "XX"

        if first_2_letters == "LG" and extension == ".PNG":
            Output_polygon_features = "D:/J040083/ShapeFiles.gdb/" + "SH_" + filename_zero
            print (Output_polygon_features)
            # Process: Raster to Polygon...

            INPUT_RASTER = os.path.join(root + "/" + filename_zero + ".png")
            print (INPUT_RASTER)

            gp.RasterToPolygon_conversion(INPUT_RASTER, Output_polygon_features)

            Distance__value_or_field_ = "5"
            Raster_Buffer_shp = "D:/J040083/ShapeFiles.gdb/" + "SB_" + filename_zero
            print (Raster_Buffer_shp)

            # Process: Buffer...
            gp.Buffer_analysis(Output_polygon_features, Raster_Buffer_shp, Distance__value_or_field_, "FULL", "ROUND", "NONE")

and i did have a problem with the files with + in them, so i made a script to replace any +s with _s!

Thank you soooo much for your help!

[V_StuartFoote]

Alice,

Good news 🙂

So now make it pretty --

1) as noted before these snippets come from saving the script out of the original model builder run and should be removed...
[INDENT]# Load required toolboxes...
##gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Conversion Tools.tbx")
##gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Analysis Tools.tbx")[/INDENT]

2) using the overwrite is a good idea,
[INDENT]gp.OverWriteOutput = 1 # overwrite existing files (1 = True, 0 = False)[/INDENT]
but I'm not certain of its behavior regards the preceding gp.CreateFileGDB ("D:/J040083", "ShapeFiles.gdb") will it clobber it or throw an error?

3) to be more "Pythonic" the os.path.join syntax could be changed
[INDENT]INPUT_RASTER = os.path.join(root, filename_zero + ".png")[/INDENT]


4) You're not working with Shapefiles, rather Polygon Feature Classes in a File Geodatabase. Since you kept the model builder naming outputs it might be better to instead rename the FGDB, i.e. BufferedFeatures.gdb or something more descriptive.

5) your buffer is FULL, do you need LEFT (outside) or RIGHT (inside) the extracted polygon features (unfortunately only available if you are on an ArcInfo level Desktop license). It is also in pixels, 5 which with FULL is doubled to a width of 10 pixels. And to work in geospatial values you may need to treat the rasters as projected during the buffering--or could project the feature class at your next analysis step.

6) the script is not especially robust as you've found. Probably need to handle more cases, e.g. files with extensions other than .PNG, or different raster types, as well as filename oddities--plus signs and periods being a problem for consistent results with Python os.path.splitext().

7) finally, you'd indicated you wanted Shapefiles as the outcome. You don't have them yet but can easily convert with a loop through the FGDB using the FeatureClassToShapefile_conversion geoprocessing command.

Otherwise congratulations! You can get on with your analysis and get to it all the faster on the next problem set.

Stuart

[AliceDuff1]

Thank you!

I have made some of the changes you suggested to make it look nice, and i am going to look into the feature class to shapefile conversion, and adding in the name change function into the script itself!

I dont think overwrite is actually doing anything in the script because it still gives an error if i dont delete the shapefile.gdb, so i will remove it!

I need the buffer to be by 5 meters either side (the .png files are pipelines, cables etc underground and need to be avoided by 5 meters either side for soil investigations!). Should i change the "5" to "5 meters", if this means 5 pixels - i assume thats not the same as 5 meters!

[V_StuartFoote]

Alice,

OK. Better role up your sleeves... 🙂

So, we haven't discussed it but what is the source of your .PNG rasters?  In any case, you'll have two choices on buffering them in a geospatial context within your GIS:

[INDENT]1) spatially georeference the raster prior to extracting the polygon/polyline buffers as feature classs by including a .WLD, .PGW, or .pngw ESRI Grid "World file" for each raster.  Once geospatialy registered you could specify a 5 meter buffer--it could be 1 pixel, 2 pixels, or even 10 pixels depending on the resolution of the raster. Each raster image will require its own unique world file. And you'll probably want to pick a "projection" with planar units in meters--like UTM.  A big down side is that your Python script will need to be modified considerably to robustly include the world files, georegister the image and apply a projection/spatial reference to be used during the raster to polygon extraction.

2) the second choice is to project the buffer feature classes, defining a projection, after they have been extracted from the raster. You'll need to establish a set of X,Y control points in a meter planar unit projection--again probably UTM, to overlay on each polygon as control for the projection. Project the points and apply the same to the buffer polygon. Do you have GPS points from your pipeline and utility data? Perhaps do the rastertopolygon buffer at .5 pixel. and then once projected, perform another buffer to be 5 meters.
[/INDENT]
The first is the more traditional approach, and is MUCH simpler to implement--but you'll have to rework your Python script.

Another suggestion would be to drop the direct RasterToPolygon based buffer geoprocessing and instead perform a RasterToPolyline feature extraction. And then buffer the polyline feature--your pipelines and other underground utilities. You'd still want to georeference and register the rasters into a planer unit projection--like UTM for accuracy of the buffer. And would need to establish "World file" details for each image, again what is the source of the PNGs--do they provide spatial reference of some sort?

An alternative to geoprocessing from the PNG would be to convert each image to GeoTIFF with embeded spatial registration, the RasterToPolygon or even RasterToPolyline with subsequent buffering could consume it.  Perhaps the latest FWTools GDAL build gdal_translate tool could be incorporated into an "os" call in the script, or done ahead of time. Maybe too much?

Stuart

[AliceDuff1]

wow!! I am not back in the office until the new year now - so i will look into all of this then and get back to you!! thank you sooo much for your help you have saved my life!!!:)

[V_StuartFoote]

No worries, and good health for the new year...

[AliceDuff1]

Stuart,

merry christmas and happy new year!

The problem i am facing is this: the .png files i have come with no other information. They are usually opened in a programme called "Klic", along with other data in .pdf format which the client needs. The client now wants these klic files (the .png files) to be imported into ArcMap where they need to be georeferenced. The only data i have for the .png files are in .xml files (i dont really grasp what these are to be honest!) But apparently they have co-ordinates in them, so I dont really have any concept of how long the pipelines are!!

I had initially tried a rastertopolyline conversion - but with some of the files that didnt work, even manually! I think it is due to the fact that some .png files are funny shapes, or that they are counting the background as a part of the shape!

How do i create a world file and attach it to the raster?

I feel so guilty for taking up so much of your time!! If you ever have a problem i can help you with (probably not) i would do the same!!!

Thank you!!!!

[V_StuartFoote · Accepted solution]

Alice,

Welcome back, fun and happy holidays?

So, before you can get back to scripting the creation of your buffers, you MUST correctly establish the projection and registration details for each PNG image you want to process.

So, I'm guessing that your Klic application is the Arcadis Klic viewer and you are dealing with PNG derived from GBKN (Grootschalige Basiskaart Nederland) projected in Dutch RD--EPSG 28992 (worked in ArcGIS as RD New coordinate system) or from other European imagery in WGS84 GCS lat/long.

In any case, the projection details you need are likely available as metadata contained in the XML and PDF files accompanying each image. The XML will be well structured and tagged, although probably not GeoXML. But you will need to study it along with any metadata in the PDF for each PNG image, and from the metadata determine where to position each image in real world coordinates. Fortunately, all the images will probably be in the same projection and the needed information will be encoded the same for all the file sets. You just have to figure out how. You may be given an upper left and lower right coordinate set, or an upper left--a pixel size--and a rotation, or even all four corners of the image.

If you are allowed to post a sample of the XML and an excerpt from the PDF along with a PNG raster and we may be able to help. But you might post that as a new thread on the Imagery and Raster Data forum to get the right folks helping to tease out the georeference details.

Then, once you assemble the registration data for the PNGs, you can either convert them to GeoTIFFs (with a GDAL script) or create an ESRI Grid World file for each.

And finally, you can get back to the geoprocessing of a 5m buffer.

You will have several options once you are able to access the georeferenced image with ArcGIS.

And you can bring each into an ArcMap feature editing session and digitize a centerline of each pipeline. Or, you can return to your gp.RasterToPolygon_conversion(), or perhaps try the RasterToPolyline_conversion() tool. The key will be that by having georeference of each image, you'll then be able to work with accurate measurements in meters rather than just image pixels.

One additional scripting option if you have ArcGIS 10 would be the new Image Classification tools for either unsupervised or supervised with training. A more sophisticated way to extract the pipeline detail out of the PNG raster and pass on for RasterToPolygon or RasterToPolyline conversion.

If someone at the office has ERDAS or ENVI image analysis suite they really are better tools for doing the same feature extractions.

Stuart