List features in multiple workspaces then convert to point, BUT ignore some paths?

[SWCASWCA]

I have this code to loop through a workspace and find all shps and feature classes that are named a certain thing. I'd like to convert each poly to a point and then append it to a feature class, BUT I want to skip over any feature that has the word "ARCHIVE" in it's path. Any tips on how to do this would be great!

BTW, the code fails so far. My final loop is merely taking the first letter of my output and doing the rest, instead of taking the actual feature class. I.e. it is telling me that "D_pt" does not exist. I know the path works, that is if I print it lists what I need it to list (all feature classes and shps that start with "APE" or "SAB") but then I can't do anything with it.

import arcpy, os  #Set the workspace workspace = r"D:/Work"  #create list of files from directories and subdirectories for root, dirs, files in arcpy.da.Walk(workspace, datatype="FeatureClass", type="Polygon"):     for name in files:         if name.startswith("APE") or name.startswith("SAB"):             path = os.path.abspath(os.path.join(root, name))             for x in path:                 outFC = x + "_pt"                 # convert to points                 arcpy.FeatureToPoint_management(path, outFC, "INSIDE") 

[RhettZufelt]

Something like this?

Hopefully, you have the same attribute names, type, etc in each of the FC's you are converting to point. Then, you would create an empty FC with the attribute table already set up and just append to it without having to worry about field mappings.

will call it appendFC for clarity.

import arcpy, os

#Set the workspace
workspace = r"D:/Work"

appendFC = r"C:\TEMP\PDX_SAB\PDX_SAB.gdb\appendFC"
outFC = r"\\in_memory\centerPoints"      # make this an in memory dataset is faster.  no need to write to disk if you are just deleting it.


#create list of files from directories and subdirectories
for root, dirs, files in arcpy.da.Walk(workspace, datatype="FeatureClass", type="Polygon"):
    for name in files:
        if name.startswith("APE") or name.startswith("SAB"):
            path = os.path.abspath(os.path.join(root, name))
            for x in path:
              if "ARCHIVE" not in x:
                                # empty the output feature class before adding new data
                if arcpy.Exists(outFC):
                    arcpy.Delete_management(outFC)  # if you are emptying it always, no need to check for size, just delete it if exists.
                # convert to points
                arcpy.FeatureToPoint_management(path, outFC, "INSIDE")
                # append to final FC
                arcpy.Append_management(outFC, appendFC)

R_

[RhettZufelt]

wasn't sure what/why you were hacking the paths and such, but after re-reading, I think you are just trying to process the polygons that start with SAB or APE if they are not in a directory named "ARCHIVE".

If this is true, try something like this:

import arcpy, os

#Set the workspace
workspace = r"D:/Work"

appendFC = r"C:\TEMP\PDX_SAB\PDX_SAB.gdb\appendFC"
outFC = r"\\in_memory\centerPoints"      # make this an in memory dataset is faster.  no need to write to disk if you are just deleting it.


#create list of files from directories and subdirectories
for dirpath, dirnames, filenames in arcpy.da.Walk(workspace,topdown=True, datatype="FeatureClass", type="Polygon"):

   if "ARCHIVE" in dirnames:
       dirnames.remove('ARCHIVE')
            
   for filename in filenames:
     if filename.startswith("APE") or filename.startswith("SAB"):
     
         if arcpy.Exists(outFC):
                arcpy.Delete_management(outFC)  # if you are emptying it always, no need to check for size, just delete it if exists.

         # convert to points
         arcpy.FeatureToPoint_management(os.path.join(dirpath,filename), outFC, "INSIDE")

         # append to final FC
         arcpy.Append_management(outFC, appendFC)   

If "ARCHIVE" is not a folder, but is included in a folder path somewhere, you could still put the if "ARCHIVE" not in code from my previous post.  If it is a folder, the way I have it coded, it should removed that directory before walking through the FCs.

R_

[SWCASWCA]

Thanks! I had just discovered the dirnames.remove feature. And yes, ARCHIVE is a subdirectory.

I am getting an error though: "ExecuteError: ERROR 000210: Cannot create output \\in_memory\centerPoints
Failed to execute (FeatureToPoint)."

import arcpy, os

#Set the workspace
in_workspace = r"D:\Projects"
appendFC = r"C:\TEMP\PDX_SAB\PDX_SAB.gdb\appendFC"
# make an in memory dataset its faster.  no need to write to disk if you are just deleting it.
outFC = r"\\in_memory\centerPoints"

#create list of files from directories and subdirectories
for dirpath, dirnames, filenames in arcpy.da.Walk(in_workspace, datatype="FeatureClass",type="Polygon"):
    # disregard any folder named "ARCHIVE" in creating list
    if "ARCHIVE" in dirnames:
        dirnames.remove('ARCHIVE')

    for filename in filenames:
        if filename.startswith("APE") or filename.startswith("SAB"):
            if arcpy.Exists(outFC):
                arcpy.Delete_management(outFC)

            # convert to points    
            arcpy.FeatureToPoint_management(os.path.join(dirpath, filename), outFC, "INSIDE")

            # append to final FC
            arcpy.Append_management(outFC, appendFC)

Also, how do I get it to print (maybe to a text file?) the paths? I want to use that to make sure I'm only appending the ones I want, in case I need to add additional exceptions (like the ARCHIVE folder). Do I just put 'print filename' after the if filename startswith line?

[RhettZufelt]

Thanks! I had just discovered the dirnames.remove feature. And yes, ARCHIVE is a subdirectory.

I am getting an error though: "ExecuteError: ERROR 000210: Cannot create output \\in_memory\centerPoints
Failed to execute (FeatureToPoint)."

Also, how do I get it to print (maybe to a text file?) the paths? I want to use that to make sure I'm only appending the ones I want, in case I need to add additional exceptions (like the ARCHIVE folder). Do I just put 'print filename' after the if filename startswith line?

My bad, try this:

outFC = "in_memory\\centerPoints"

yes, print filename after the startswith line at the same indentation level as the if ArcpyExists to print to the console.

could put print os.path.join(dirpath, filename) if you want the entire path/filename.

To print to a text file, you need to open it in write mode and write to it.  Here is an example of some of my code that writes results to a text file.

with open('D:/update/temp.txt','a',0) as f:         ### the "a" tells it to append to existing file.  "w" for write would create a new file, even if it exists.

  print dt
  f.write(dt)
  f.write("\n\nthis is first line of image_tables.py \n\n")

 # Process: Delete


  if arcpy.Exists(Images_full):
     print "deleting ",Images_full
     f.write("\ndeleting ")
     f.write(Images_full)
     arcpy.Delete_management(Images_full, "Table")

  if arcpy.Exists(WasteSitePoly_centroids):
     print "deleting ",WasteSitePoly_centroids
     f.write("\ndeleting ")
     f.write(WasteSitePoly_centroids)
     arcpy.Delete_management(WasteSitePoly_centroids)



f.close    # at the same indentation level as the with open line

Since I opened it as "f"  (with open as f), then f.write are the statements that write to the file, the print just echos to the screen, so, what is printed to the screen is also written to the text file.
a print statement automatically indexes to a new line when you hit enter, the f.write statements, the \n adds a new line, otherwise, it just concatenates one long line.

R_

[SWCASWCA]

Again, many thanks for the code! Is there a particular place I have to put the write to text file code? I imagine the f.write line has to go after the print statement but what about all the stuff at the beginning?

So I think it's working, but I did forget that I am not keeping any of the fields once I've converted to a point. I do want to capture the original path, and also a small portion of the path, and calc them to two attribute fields in the appendFC.

So for example, for the following paths:

D:\Projects\12345_XXX\SHP\APE.shp
D:\Projects\34567_XXX\GDB\MyGDB.gdb\SAB

I would want two records in my appendFC where the ProjNum field has been calc'd to be 12345 and 34567 respectively. I'm trying to capture it with this code but it isn't working.

import arcpy, os

#Set the workspace
in_workspace = r"D:\Projects"
appendFC = r"C:\TEMP\PDX_SAB\PDX_SAB.gdb\PDX_SAB_Centerpoint"
# make an in memory dataset its faster.  no need to write to disk if you are just deleting it.
outFC = "in_memory\\centerPoints"

# create list of files from directories and subdirectories
for dirpath, dirnames, filenames in arcpy.da.Walk(in_workspace, datatype="FeatureClass",type="Polygon"):
    # disregard any folder named "ARCHIVE" in creating list
    if "ARCHIVE" in dirnames:
        dirnames.remove('ARCHIVE')

    for filename in filenames:
        if filename.startswith("APE") or filename.startswith("SAB"):
            print os.path.join(dirpath, filename)
            if arcpy.Exists(outFC):
                arcpy.Delete_management(outFC)

            # convert to points    
            arcpy.FeatureToPoint_management(os.path.join(dirpath, filename), outFC, "INSIDE")

            # add field to capture project number
            arcpy.AddField_management(outFC, "ProjectNum", "TEXT", "", "", 20, "", "NULLABLE", "REQUIRED")
            arcpy.AddField_management(outFC, "OriginalPath", "TEXT", "", "", 255, "", "NULLABLE", "REQUIRED")

            # extract ProjectNum from path and calc fields
            exp = str(os.path.join(dirpath, filename)[9:13])
            
            arcpy.CalculateField_management(outFC, "ProjectNum", os.path.join(dirpath, filename), '"' + exp + '"', "PYTHON")
            arcpy.CalculateField_management(outFC, "OriginalPath", os.path.join(dirpath, filename), "PYTHON")
            
            # delete fields; use ListFields to get a list of field objects
            fieldObjList = arcpy.ListFields(outFC)
 
            # Create an empty list that will be populated with field names        
            fieldNameList = []
 
            # For each field in the object list, add the field name to the
            #  name list.  If the field is required, exclude it, to prevent errors
            for field in fieldObjList:
                if not field.required:
                    fieldNameList.append(field.name)

            # delete field
            arcpy.DeleteField_management(outFC, fieldNameList)

            # append to final FC
            arcpy.Append_management(outFC, appendFC)

I'm getting this error:
Traceback (most recent call last):
  File "C:/TEMP/PDX_SAB/SAB_Python.py", line 31, in <module>
    arcpy.CalculateField_management(outFC, "ProjectNum", os.path.join(dirpath, filename), '"' + exp + '"', "PYTHON")
  File "C:\Program Files (x86)\ArcGIS\Desktop10.1\arcpy\arcpy\management.py", line 3129, in CalculateField
    raise e
ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000800: The value is not a member of VB | PYTHON | PYTHON_9.3.
Failed to execute (CalculateField).

[RhettZufelt · Accepted solution]

the error is related to your expression.  Often I find it is easier to create the expression string as a variable and pass this way.
this is working for me:

import arcpy, os  #Set the workspace in_workspace = r"D:\Projects" appendFC = r"C:\TEMP\PDX_SAB\PDX_SAB.gdb\PDX_SAB_Centerpoint" # make an in memory dataset its faster.  no need to write to disk if you are just deleting it. outFC = "in_memory\\centerPoints"            with open(r'D:/Projects/temp.txt','w',0) as f:       # create list of files from directories and subdirectories     for dirpath, dirnames, filenames in arcpy.da.Walk(in_workspace, datatype="FeatureClass",type="Polygon"):         # disregard any folder named "ARCHIVE" in creating list         if "ARCHIVE" in dirnames:             dirnames.remove('ARCHIVE')          for filename in filenames:             if filename.startswith("APE") or filename.startswith("SAB"):                 print os.path.join(dirpath, filename)                 f.write(os.path.join(dirpath, filename))                 f.write("\n")                 if arcpy.Exists(outFC):                     arcpy.Delete_management(outFC)                  # convert to points                     arcpy.FeatureToPoint_management(os.path.join(dirpath, filename), outFC, "INSIDE")                  # add field to capture project number                 arcpy.AddField_management(outFC, "ProjectNum", "TEXT", "", "", 20, "", "NULLABLE", "REQUIRED")                 arcpy.AddField_management(outFC, "OriginalPath", "TEXT", "", "", 255, "", "NULLABLE", "REQUIRED")                  # extract ProjectNum from path and calc fields                 exp = str(os.path.join(dirpath, filename)[9:13])                 exp2 = str(os.path.join(dirpath, filename)                  arcpy.CalculateField_management(outFC, "ProjectNum","exp", "PYTHON_9.3")                 arcpy.CalculateField_management(outFC, "OriginalPath","exp2" ,"PYTHON_9.3")                  # delete fields; use ListFields to get a list of field objects                 fieldObjList = arcpy.ListFields(outFC)                  # Create an empty list that will be populated with field names                         fieldNameList = []                  # For each field in the object list, add the field name to the                 #  name list.  If the field is required, exclude it, to prevent errors                 for field in fieldObjList:                     if not field.required:                         fieldNameList.append(field.name)                  # delete field                 arcpy.DeleteField_management(outFC, fieldNameList)                  # append to final FC                 arcpy.Append_management(outFC, appendFC)      f.write("done with script \n") f.close

I took the liberty to add the file writes to your script.  Basically, after the with open statement, the file is open for writing as "f".
so, anytime between the with open, and the f.close if you do a f.write() statement, it will write it to the text file.  f.close has to be at same indentation as the with open.
You now have example of how to write a variable, a new row, and text (with new row).

R_

Also, do you use modelbuider?  For this kind of script, my initial approach would be to create a model in arcmap to perform this on just one FC and not worry about iterating.

So, in the model, I would pull in the FeatureToPoint tool and convert to in_memory FC
then I'd pull in two AddField tools and configure to create both the fields
then pull in two calculateField tools and configure both
then would pull in the Append tool and configure to append to the appendFC.

now, run the model and make sure it works.  If you get errors, fix them until the model runs correctly.
Now, File menu, export, export to python script.

Now you have a starting point, with the proper syntax for most the tools and all you have to do is add the da.walk iterator stuff to make it work on multiples.

[SWCASWCA]

You are right - I went about this sort of backwards. I'm trying to learn more Python and thought the iteration was the place to start (just finding and 'gathering' all the data I needed to work with before doing anything with it). Your suggestion to start with model builder is a good one.

Thanks for the write to text code too! That is all new to me.

Can you tell me why you switched from "PYTHON" to "PYTHON_9.3" in the Calc Field expression? Is that so you can use "exp" without having to escape the quotation marks? From reading other posts I was under the impression that because exp and exp2 were strings they couldn't be used as variables without formatting them as '"' + exp + '"'.

Even using model builder I still can't get the expression right to calc the path to a field. I tried the parse path but either I'm not using it right or that is not it's intention.

Back in the python script I have a feeling the script is not running in the order I specified. I keep getting an error message about the schemas not matching to do the append. When I try printing the fields after the delete statement I see all the original fields - meaning no new fields were added and calc'd, and the old fields were not deleted.

How do I ensure that the processes run in the order I've put them in the script?

[RhettZufelt]

Can you tell me why you switched from "PYTHON" to "PYTHON_9.3" in the Calc Field expression? Is that so you can use "exp" without having to escape the quotation marks? From reading other posts I was under the impression that because exp and exp2 were strings they couldn't be used as variables without formatting them as '"' + exp + '"'.

Because it seems that is the one I can ususally get to work as expected. Guess I'm getting used to haveing objects returned rather than strings.

â?¢To calculate strings to text or character fields, in the dialog box the string must be double-quoted ("string"),   or in scripting, the double-quoted string must also be encapsulated in single quotes ('"string"').

Otherwise, just encapulated in a single set of double quotes passes the variable. Not sure where the + exp + comes from.

Basically, the expression needs the quote around it when it gets passed to calcfield. If you didn't put the double quotes around exp2 in the calcfiled line, you would have to include them in the exp2 string, and since that is a calculation, would have to concatenate them on there. Easier to just pass them around the exp variable.

exp2 = '"' + str(os.path.join(dirpath, filename)) + '"'

If you used that for your exp2 variable, it would include the double quotes, so in the calcfield you would just put exp2 without the double quotes.
Hope this makes sense.

Even using model builder I still can't get the expression right to calc the path to a field. I tried the parse path but either I'm not using it right or that is not it's intention.

so, what are you getting for the print os.path.join(dirpath, filename), or what is getting written to the text file for this value? The way I have it, it will return my path u'd:\\working.gdb\dataset' which I wrap in a str() for my exp2 variable. Oh heck, now I see an error in my code that could be causing this (breaking it for sure).

Change this:
exp2 = str(os.path.join(dirpath, filename)

to this:
exp2 = str(os.path.join(dirpath, filename))

After that, see if it works, if not, let me know what the print statements say. Actually, the one that matters is, what does

print exp2 report?

Also, noticed you have set the field lenth to 255, are you sure your path(s) are not exceeding that length? In ArcMap, it will warn you and truncate, not sure how it handles that in code.

Back in the python script I have a feeling the script is not running in the order I specified. I keep getting an error message about the schemas not matching to do the append. When I try printing the fields after the delete statement I see all the original fields - meaning no new fields were added and calc'd, and the old fields were not deleted. 

How do I ensure that the processes run in the order I've put them in the script? 

Unless you are redirecting it somewhere (sending it to a def()), it will run line by line, so the order is the same. You are not redirecting, so it will do line by line, except, of course where you loop through using for statements.

I suspect that you are running into this issue for append:

Specifies if the schema (field definitions) of the input datasets must match the schema of the target dataset in order for data to be appended. 

â?¢TEST â??Input dataset schema (field definitions) must match the schema of the target dataset. An error will be returned if the schemas do not match.  
â?¢NO_TEST â??Input dataset schema (field definitions) do not have to match that of the target dataset. Any fields from the input datasets that do not match the fields of the target dataset will not be mapped to the target dataset unless the mapping is explicitly set in the Field Map control. 

First, have you ensured that there is an existing point FC named C:\TEMP\PDX_SAB\PDX_SAB.gdb\PDX_SAB_Centerpoint that has two text fields (and is in writeable workspace, and not being viewed in ArcMap or Catalog by anyone (otherwise, lock file..)).

One named ProjectNum with size 20
and
one named OriginalPath with size 255


And then try this for you append line

arcpy.Append_management(outFC, appendFC, "NO_TEST", "", "")

see if that gets past the schema issues. With the NO_TEST, it should copy the attributes that match exactly.

R_

[SWCASWCA]

Well, I ended up figuring it out and as is usually the case with code (that I've found) it all comes down to syntax: quotes, double quotes, = vs ==, etc. I gave up using model builder, as I couldn't figure out the error messages I was getting. I found the errors in the python script to be much easier to work through.

Thanks again for all your invaluable help! Now on to tackling why it creates multiple points each time . . .

Final code:

import arcpy, os

#Set the workspace
in_workspace = r"C:\TEMP\PDX_SAB"
appendFC = r"C:\TEMP\PDX_SAB\PDX_SAB.gdb\PDX_SAB_Centerpoint"
# make an in memory dataset its faster.  no need to write to disk if you are just deleting it.
outFC = "in_memory\\centerPoints"

with open(r'C:\TEMP\PDX_SAB\paths.txt','w',0) as f:

    # create list of files from directories and subdirectories
    for dirpath, dirnames, filenames in arcpy.da.Walk(in_workspace, datatype="FeatureClass",type="Polygon"):
        # disregard any folder named "ARCHIVE" in creating list
        if "ARCHIVE" in dirnames:
            dirnames.remove('ARCHIVE')

        for filename in filenames:
            if filename.startswith("APE") or filename.startswith("SAB"):
            #if filename.name == "APE" or filename.name == "SAB":
                #print os.path.join(dirpath, filename)
                f.write(os.path.join(dirpath, filename))
                f.write("\n")
                if arcpy.Exists(outFC):
                    arcpy.Delete_management(outFC)

                # convert to points    
                arcpy.FeatureToPoint_management(os.path.join(dirpath, filename), outFC, "INSIDE")

                # add field to capture project number
                arcpy.AddField_management(outFC, "ProjectNum", "TEXT", "", "", 20, "", "NULLABLE", "REQUIRED")
                arcpy.AddField_management(outFC, "OriginalPath", "TEXT", "", "", 255, "", "NULLABLE", "REQUIRED")
                arcpy.AddField_management(outFC, "ServiceArea", "TEXT", "", "", 75, "", "NULLABLE", "REQUIRED")

                # set expressions to calculate fields
                exp = str(os.path.join(dirpath, filename)[16:21])
                exp2 = str(os.path.join(dirpath, filename))
                if filename.startswith("APE"):
                    exp3 = "Cultural"
                else:
                    exp3 = "Natural"
            
                arcpy.CalculateField_management(outFC, "ProjectNum", "exp", "PYTHON_9.3")
                arcpy.CalculateField_management(outFC, "OriginalPath", "exp2", "PYTHON_9.3")
                arcpy.CalculateField_management(outFC, "ServiceArea", "exp3", "PYTHON_9.3")
            
                # delete fields; use ListFields to get a list of field objects
                fieldObjList = arcpy.ListFields(outFC)
 
                # Create an empty list that will be populated with field names        
                fieldNameList = []
 
                # For each field in the object list, add the field name to the
                # name list.  If the field is required, exclude it, to prevent errors
                for field in fieldObjList:
                    if not field.required:
                    #if field.name != "ProjectNum" and field.name != "OriginalPath":
                        fieldNameList.append(field.name)

                # delete field
                arcpy.DeleteField_management(outFC, fieldNameList)

                # append to final FC
                arcpy.Append_management(outFC, appendFC)
    f.write("done with script \n")
f.close