Select Layer by Location not providing correct result.

[SusanWhitney]

I sought assistance with a portion of this code earlier. I have added an additional step, which to me would seem pretty simple, but I am still learning the ups & downs of Python.

My script creates data driven pages. I use a list to populate a layer definition query to run a specific township. Now I am trying to do a selection (SelectLayerByLocation) to only find the townships where there is data. I first just added the SelectLayerByLocation command. Did not work. With further reading, it looked like I needed to run a MakeFeatureLayer on the two sets of data involved in the selection. Still did not work. Do I need to use cursors or is the answer much simpler?

I do not have any error messages. I have added try/except to see if I can find additional information on what is happening with my script. The data exports just fine, but there are maps that do not have any data.
Thank you for any guidance.

# PoleDDpages.py # Author: slw # Date: XXXXX # Revisions: XXXXX # Description: Create Pole map book series using Data Driven Pages and #           insert a cover page  # Import ArcPy import os,arcpy  try:          # Set up variables     # Location of pole map .mxd document     mxdDoc = r"G:\GEOSPATIAL\Publishing\Pole\Pole_QtrSec.mxd"      # Create the MapDocument object     mxd = arcpy.mapping.MapDocument(mxdDoc)     df = arcpy.mapping.ListDataFrames(mxd)[0]     layer = arcpy.mapping.ListLayers (mxd, "SURVEY_GRID_BNDRY",df)[0]     poleLyr = arcpy.mapping.ListLayers (mxd, "Pole",df)[0]      # Overwrite existing file     arcpy.env.overwriteOutput = True      # Output directory for the pole maps     outDir = r"G:\GEOSPATIAL\Publishing\Pole"      # Set the workspace     arcpy.env.workspace = outDir      # List of map grids     twpList = ['\"TOWNSHIP_C\" = \'D5\' AND \"RANGE_CD\" = \'5\'',\                '\"TOWNSHIP_C\" = \'D6\' AND \"RANGE_CD\" = \'4\'']       i=1     for twpName in twpList:         layer.definitionQuery = twpName          # refresh() after changing the layer def query to 'redefine' the DDP index limits         mxd.dataDrivenPages.refresh()                 # Create temporary layers to work with the Selection         arcpy.MakeFeatureLayer_management(layer, "surveyLyr")         arcpy.MakeFeatureLayer_management(poleLyr, "poleLyr_new")          # Clear the Selection before starting         #arcpy.SelectLayerByAttribute_management("surveyLyr", "CLEAR_SELECTION", "")         #print "The selection has been cleared"                  # Select Layer By Location to limit to just maps with data         arcpy.SelectLayerByLocation_management("surveyLyr", "INTERSECT", "poleLyr_new", "", "NEW_SELECTION")          # refresh() after changing the layer def query to 'redefine' the DDP index limits         mxd.dataDrivenPages.refresh()          mxd.saveACopy(os.path.splitext(mxdDoc) [0] + str(i) + os.path.splitext(mxdDoc)[1])         i += 1                   # The final mapbook name taken from the list         finalPDFfn = outDir + "\\" + twpName [16:18] + twpName [38:39] + "Pole.pdf"                  # Create the final PDF -- which is just an empty shell right now         finalPDF = arcpy.mapping.PDFDocumentCreate(finalPDFfn)          # A temporary pdf file for processing         tmpPDF = outDir + "\\PoleMapPages.pdf"          # Let the user know what is happening!         print "Exporting " + twpName [16:18] + twpName [38:39]                  # Export the data driven pages in the mxd to a temporary PDF         print "Exporting map pages to the temporary PDF"         ddp = mxd.dataDrivenPages.exportToPDF(tmpPDF)          # Append the temporary pdf to the final pdf         # Cover, map pages, back cover         print "Appending Map Pages"         finalPDF.appendPages (r"G:\GEOSPATIAL\Publishing\Pole\PoleCovers\Covers_"\                               + twpName [16:18] + twpName [38:39] + ".pdf")         finalPDF.appendPages(tmpPDF)         finalPDF.appendPages(r"G:\GEOSPATIAL\Publishing\TwpGrid_Color8x11.pdf")          # Set properties for Adobe Reader and save PDF.         finalPDF.updateDocProperties(pdf_open_view = "USE_THUMBS", pdf_layout = "SINGLE_PAGE")         finalPDF.saveAndClose()          # Deleting temporary layers         arcpy.Delete_management("surveyLyr")         arcpy.Delete_management("poleLyr_new")      except Exception as e:     print e.message     print arcpy.GetMessages(2)      # Clean up print "Cleaning up" # Delete the temporary PDF using the ArcPy function if arcpy.Exists(tmpPDF):                  arcpy.Delete_management(tmpPDF)   # Delete objects del mxd, tmpPDF, ddp               # Finished message print "Map compilation completed. Please review the final output."

[MikeMacRae]

The only thing I see is in your List, it seems like there might be an extra back slash between your 2 items. The one after your comma. Should this be there? If I am right, then the definition query is returning nothing and therefore selecting nothing.

twpList = ['\"TOWNSHIP_C\" = \'D5\' AND \"RANGE_CD\" = \'5\'',\
               '\"TOWNSHIP_C\" = \'D6\' AND \"RANGE_CD\" = \'4\'']

Also, I really like the Traceback error handling code to better identify the first line in error. maybe there is something else not firing along the way before it gets to that. Give it a shot:

http://resources.arcgis.com/en/help/main/10.1/index.html#//002z0000000q000000

[SusanWhitney]

Thank you, I'll try adding the Traceback to see if I get any information.

The definition query works correctly. I got that to work before I added the selection statement. The extra backslash was added so I could put the next 'item' on a new line. It just lets python know that the line continues.
Thank you

[T__WayneWhitley]

I think you should have to refresh the DDP after you set the def query...and the reason for that is you need to make the DDP function 'aware' of the change you made to the index layer, because it generates pages based on it --- kind of similar to refreshing a cache, if you know what I mean?  (I added that about the cache to help in a 'conceptual' way.)

So, once DDP 'understands' what it is to export, I am not quite sure the need to 're-refresh', if you'll pardon the term?  Even if you had to refresh once just before exporting, that should be sufficient.  If you would please, clarify your select or query operation you intend to do in-between the def query set on the index layer (which I know runs fine) and your export to PDF?


Thanks,
Wayne

EDIT:  Tell me please the reason you need to redefine what is visible in your layers to match the index layer which itself has been redefined?  I cannot 'see' what is going on, but it doesn't seem necessary if your redefined index layer by design will only show what is on those pages.

Are you just trying to filter out unanticipated 'no data' pages?

1:50 PM-
Okay, I think I understand now and see what is wrong - you are trying to further refine the index layer, but you haven't futher manipulated the def query on that layer, so the 2nd 'refresh' command is meaningless.  I think you need to get the sections where there is no data (if that is what you're trying to further limit your index layer with) and add to the def query, reset it on the index layer, and then run your refresh again.

Make sense now?  ...something like this logic (but of course make it 'grammatically' correct):

layer.definitionQuery = twnName + 'and not in [twn1, twn2, twn3, etc....]'

...where twn1, twn2, twn3 and so forth represent the townships of no data extracted from your select by location results.  Then refresh and your 'layer' should be made aware...  I think you're on the right track.

[SusanWhitney]

Good morning,
Definition query defines the township/range to process and the select by location selects only the sections and quarter sections in the township/range that do not have data.

So are you saying to not use the select by location and set it up as a separate definition query or do the select by location first and then the definition query.

I appreciate your assistance. This scripting is my trek into unknown waters.

[T__WayneWhitley]

Susan, I do not know why this didn't dawn on me before, but I think you could do this, provided I have assumed correctly you want to be 'sensitive' to how your pole layer has grown within your grid index layer:

1- Before the loop on your query, run Select By Location (intersect is fine) the grid index layer with the pole layer.

The result should be only grid index features where you have pole layer features.  Consider this a 'prefilter' or intermediate result - you were trying to run this in your loop which is not only unnecessary but more processing intensive as well.

2- Copy features to a temp gdb, overwriting as necessary (think you already have this set).  Have a lyr file sourced to this temp fc.  [or, if you like, I think you could directly source your index layer to the temp fc]

3- Use UpdateLayer to 'get' the lyr file to update your index layer in your map (from the dynamically 'prefiltered' temp gdb fc of your grid index layer). [This step won't be necessary if you have replaced the datasource as in step 2]

4- Do your loop to then filter on section/ranges with your query and export pdfs...this way, you only need to refresh the index once for each loop.

That clear?  Hope that clears things up, and essentially you were doing things 'backwards'.  Also, you could do it via the def query as I suggested earlier, but think that would be more processing intensive as well.


Hope that helps.

-Wayne

[T__WayneWhitley]

Susan - this is your modified code going the modified def query route I initially suggested.  Not sure which is the better performer, however this should not require any extra layers or use of a temp gdb - please try it, as it has not been tested, and let me know if you get any errors:

# PoleDDpages.py
# Author: slw
# Date: XXXXX
# Revisions: XXXXX
# Description: Create Pole map book series using Data Driven Pages and
#           insert a cover page

# Import ArcPy
import os,arcpy

# Overwrite existing file
arcpy.env.overwriteOutput = True

try:
    
    # Set up variables
    # Location of pole map .mxd document
    mxdDoc = r"G:\GEOSPATIAL\Publishing\Pole\Pole_QtrSec.mxd"

    # Create the MapDocument object
    mxd = arcpy.mapping.MapDocument(mxdDoc)
    df = arcpy.mapping.ListDataFrames(mxd)[0]
    layer = arcpy.mapping.ListLayers (mxd, "SURVEY_GRID_BNDRY",df)[0]
    poleLyr = arcpy.mapping.ListLayers (mxd, "Pole",df)[0]

    ################
    # moved part of code to select grids (layer) intersecting 'Pole' features (poleLyr)

    # Create temporary layers to work with the Selection
    arcpy.MakeFeatureLayer_management(layer, "surveyLyr")
    arcpy.MakeFeatureLayer_management(poleLyr, "poleLyr_new")

    arcpy.SelectLayerByAttribute_management("surveyLyr", "CLEAR_SELECTION")
        
    # Select Layer By Location to limit to just maps with data
    arcpy.SelectLayerByLocation_management("surveyLyr", "INTERSECT", "poleLyr_new", "", "NEW_SELECTION")

    # Susan, check that this works, switching the selection...
    arcpy.SelectLayerByAttribute_management("surveyLyr", "SWITCH_SELECTION")

    # new code to get at OBJECTIDs to modify def query
    IDs = []
    rows = arcpy.SearchCursor("surveyLyr")
    for row in rows:
          newID = row.OBJECTID
          if newID not in IDs:
               IDs.append(newID)

    del row, rows

    subquery = '('
    for each in IDs:
         subquery = subquery + str(each) + ','

    subquery = subquery[0:-1] + ')'
    ################

    # Output directory for the pole maps
    outDir = r"G:\GEOSPATIAL\Publishing\Pole"

    # Set the workspace
    arcpy.env.workspace = outDir

    # List of map grids
    twpList = ['\"TOWNSHIP_C\" = \'D5\' AND \"RANGE_CD\" = \'5\'',\
               '\"TOWNSHIP_C\" = \'D6\' AND \"RANGE_CD\" = \'4\'']


    i=1
    for twpName in twpList:
        # modified the definitionQuery to include the subquery exclusion clause
        layer.definitionQuery = twpName + " AND \"OBJECTID\" NOT IN " + subquery

        # refresh() after changing the layer def query to 'redefine' the DDP index limits
        mxd.dataDrivenPages.refresh()
       
        mxd.saveACopy(os.path.splitext(mxdDoc) [0] + str(i) + os.path.splitext(mxdDoc)[1])
        i += 1
         
        # The final mapbook name taken from the list
        finalPDFfn = outDir + "\\" + twpName [16:18] + twpName [38:39] + "Pole.pdf"
        
        # Create the final PDF -- which is just an empty shell right now
        finalPDF = arcpy.mapping.PDFDocumentCreate(finalPDFfn)

        # A temporary pdf file for processing
        tmpPDF = outDir + "\\PoleMapPages.pdf"

        # Let the user know what is happening!
        print "Exporting " + twpName [16:18] + twpName [38:39]
        
        # Export the data driven pages in the mxd to a temporary PDF
        print "Exporting map pages to the temporary PDF"
        ddp = mxd.dataDrivenPages.exportToPDF(tmpPDF)

        # Append the temporary pdf to the final pdf
        # Cover, map pages, back cover
        print "Appending Map Pages"
        finalPDF.appendPages (r"G:\GEOSPATIAL\Publishing\Pole\PoleCovers\Covers_"\
                              + twpName [16:18] + twpName [38:39] + ".pdf")
        finalPDF.appendPages(tmpPDF)
        finalPDF.appendPages(r"G:\GEOSPATIAL\Publishing\TwpGrid_Color8x11.pdf")

        # Set properties for Adobe Reader and save PDF.
        finalPDF.updateDocProperties(pdf_open_view = "USE_THUMBS", pdf_layout = "SINGLE_PAGE")
        finalPDF.saveAndClose()

        # Deleting temporary layers
        arcpy.Delete_management("surveyLyr")
        arcpy.Delete_management("poleLyr_new")    

except Exception as e:
    print e.message
    print arcpy.GetMessages(2)
    
# Clean up
print "Cleaning up"
# Delete the temporary PDF using the ArcPy function
if arcpy.Exists(tmpPDF):         
        arcpy.Delete_management(tmpPDF)  
# Delete objects
del mxd, tmpPDF, ddp             

# Finished message
print "Map compilation completed. Please review the final output."

[SusanWhitney]

No errors, Wayne, but I still have the map sheets that do not have any data.

[T__WayneWhitley]

To be specific, the code I last posted runs without error and the exported map sheets reflect some index pages without pole data?

Immediately after the def query was refreshed, as a check, the following command executed in your loop:
mxd.saveACopy(os.path.splitext(mxdDoc) [0] + str(i) + os.path.splitext(mxdDoc)[1])

...and should have written the 2 mxd documents:
G:\GEOSPATIAL\Publishing\Pole\Pole_QtrSec1.mxd
G:\GEOSPATIAL\Publishing\Pole\Pole_QtrSec2.mxd

In those documents, can you check the index layer def query and the consequent DDP to see if the refresh of those pages has or has not taken place?  That will help isolate the error.  I suspect if the def query is not in the index layer's properties, then the Select By Location operation did not go as planned.  If you will, just copy/paste the different def queries here.  If that went well, then I know to look at the pdf export portion of the code.

Enjoy,
Wayne

[T__WayneWhitley]

I did discover an error in the the part I added to the def query - forgot to convert to string and got this error:

TypeError: cannot concatenate 'str' and 'tuple' objects

The error occurs on adding 'subquery' which is not a string:

layer.definitionQuery = twpName + " AND \"OBJECTID\" NOT IN " + subquery

...so the fix is to add the conversion, simply add 'str' to the line as follows:

layer.definitionQuery = twpName + " AND \"OBJECTID\" NOT IN " + str(subquery)

I did not test the rest of your code, but it seems you should look into better error trapping too.

Hope that helps - run it again, and if for some reason the PDFs aren't right, follow the instructions on my last post to check the test mxds to let me know at least the definition queries are working.  After we know everything is working, you shouldn't need that line anymore to print the mxds - you should be printing your final output from the 'parent' DDP document you've set in the beginning.

Enjoy,
Wayne