topic Re: List features in multiple workspaces then convert to point, BUT ignore some paths in Python Questions

List features in multiple workspaces then convert to point, BUT ignore some paths?

SWCASWCA — Wed, 31 Jul 2013 17:34:11 GMT

I have this code to loop through a workspace and find all shps and feature classes that are named a certain thing. I'd like to convert each poly to a point and then append it to a feature class, BUT I want to skip over any feature that has the word "ARCHIVE" in it's path. Any tips on how to do this would be great!

BTW, the code fails so far. My final loop is merely taking the first letter of my output and doing the rest, instead of taking the actual feature class. I.e. it is telling me that "D_pt" does not exist. I know the path works, that is if I print it lists what I need it to list (all feature classes and shps that start with "APE" or "SAB") but then I can't do anything with it.

import arcpy, os  #Set the workspace workspace = r"D:/Work"  #create list of files from directories and subdirectories for root, dirs, files in arcpy.da.Walk(workspace, datatype="FeatureClass", type="Polygon"):     for name in files:         if name.startswith("APE") or name.startswith("SAB"):             path = os.path.abspath(os.path.join(root, name))             for x in path:                 outFC = x + "_pt"                 # convert to points                 arcpy.FeatureToPoint_management(path, outFC, "INSIDE")

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

RhettZufelt — Sun, 12 Dec 2021 16:10:24 GMT

Something like this?

Hopefully, you have the same attribute names, type, etc in each of the FC's you are converting to point. Then, you would create an empty FC with the attribute table already set up and just append to it without having to worry about field mappings.

will call it appendFC for clarity.

import arcpy, os

#Set the workspace
workspace = r"D:/Work"

appendFC = r"C:\TEMP\PDX_SAB\PDX_SAB.gdb\appendFC"
outFC = r"\\in_memory\centerPoints"      # make this an in memory dataset is faster.  no need to write to disk if you are just deleting it.


#create list of files from directories and subdirectories
for root, dirs, files in arcpy.da.Walk(workspace, datatype="FeatureClass", type="Polygon"):
    for name in files:
        if name.startswith("APE") or name.startswith("SAB"):
            path = os.path.abspath(os.path.join(root, name))
            for x in path:
              if "ARCHIVE" not in x:
                                # empty the output feature class before adding new data
                if arcpy.Exists(outFC):
                    arcpy.Delete_management(outFC)  # if you are emptying it always, no need to check for size, just delete it if exists.
                # convert to points
                arcpy.FeatureToPoint_management(path, outFC, "INSIDE")
                # append to final FC
                arcpy.Append_management(outFC, appendFC)

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

RhettZufelt — Sat, 11 Dec 2021 09:54:01 GMT

wasn't sure what/why you were hacking the paths and such, but after re-reading, I think you are just trying to process the polygons that start with SAB or APE if they are not in a directory named "ARCHIVE".

If this is true, try something like this:

import arcpy, os

#Set the workspace
workspace = r"D:/Work"

appendFC = r"C:\TEMP\PDX_SAB\PDX_SAB.gdb\appendFC"
outFC = r"\\in_memory\centerPoints"      # make this an in memory dataset is faster.  no need to write to disk if you are just deleting it.


#create list of files from directories and subdirectories
for dirpath, dirnames, filenames in arcpy.da.Walk(workspace,topdown=True, datatype="FeatureClass", type="Polygon"):

   if "ARCHIVE" in dirnames:
       dirnames.remove('ARCHIVE')
            
   for filename in filenames:
     if filename.startswith("APE") or filename.startswith("SAB"):
     
         if arcpy.Exists(outFC):
                arcpy.Delete_management(outFC)  # if you are emptying it always, no need to check for size, just delete it if exists.

         # convert to points
         arcpy.FeatureToPoint_management(os.path.join(dirpath,filename), outFC, "INSIDE")

         # append to final FC
         arcpy.Append_management(outFC, appendFC)

If "ARCHIVE" is not a folder, but is included in a folder path somewhere, you could still put the if "ARCHIVE" not in code from my previous post. If it is a folder, the way I have it coded, it should removed that directory before walking through the FCs.

R_

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

SWCASWCA — Sat, 11 Dec 2021 09:54:04 GMT

Thanks! I had just discovered the dirnames.remove feature. And yes, ARCHIVE is a subdirectory.

I am getting an error though: "ExecuteError: ERROR 000210: Cannot create output \\in_memory\centerPoints
Failed to execute (FeatureToPoint)."

import arcpy, os

#Set the workspace
in_workspace = r"D:\Projects"
appendFC = r"C:\TEMP\PDX_SAB\PDX_SAB.gdb\appendFC"
# make an in memory dataset its faster.  no need to write to disk if you are just deleting it.
outFC = r"\\in_memory\centerPoints"

#create list of files from directories and subdirectories
for dirpath, dirnames, filenames in arcpy.da.Walk(in_workspace, datatype="FeatureClass",type="Polygon"):
    # disregard any folder named "ARCHIVE" in creating list
    if "ARCHIVE" in dirnames:
        dirnames.remove('ARCHIVE')

    for filename in filenames:
        if filename.startswith("APE") or filename.startswith("SAB"):
            if arcpy.Exists(outFC):
                arcpy.Delete_management(outFC)

            # convert to points    
            arcpy.FeatureToPoint_management(os.path.join(dirpath, filename), outFC, "INSIDE")

            # append to final FC
            arcpy.Append_management(outFC, appendFC)

Also, how do I get it to print (maybe to a text file?) the paths? I want to use that to make sure I'm only appending the ones I want, in case I need to add additional exceptions (like the ARCHIVE folder). Do I just put 'print filename' after the if filename startswith line?

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

RhettZufelt — Sat, 11 Dec 2021 09:54:07 GMT

Thanks! I had just discovered the dirnames.remove feature. And yes, ARCHIVE is a subdirectory.

I am getting an error though: "ExecuteError: ERROR 000210: Cannot create output \\in_memory\centerPoints
Failed to execute (FeatureToPoint)."

Also, how do I get it to print (maybe to a text file?) the paths? I want to use that to make sure I'm only appending the ones I want, in case I need to add additional exceptions (like the ARCHIVE folder). Do I just put 'print filename' after the if filename startswith line?

My bad, try this:

outFC = "in_memory\\centerPoints"

yes, print filename after the startswith line at the same indentation level as the if ArcpyExists to print to the console.

could put print os.path.join(dirpath, filename) if you want the entire path/filename.

To print to a text file, you need to open it in write mode and write to it. Here is an example of some of my code that writes results to a text file.

with open('D:/update/temp.txt','a',0) as f:         ### the "a" tells it to append to existing file.  "w" for write would create a new file, even if it exists.

  print dt
  f.write(dt)
  f.write("\n\nthis is first line of image_tables.py \n\n")

 # Process: Delete


  if arcpy.Exists(Images_full):
     print "deleting ",Images_full
     f.write("\ndeleting ")
     f.write(Images_full)
     arcpy.Delete_management(Images_full, "Table")

  if arcpy.Exists(WasteSitePoly_centroids):
     print "deleting ",WasteSitePoly_centroids
     f.write("\ndeleting ")
     f.write(WasteSitePoly_centroids)
     arcpy.Delete_management(WasteSitePoly_centroids)



f.close    # at the same indentation level as the with open line

Since I opened it as "f" (with open as f), then f.write are the statements that write to the file, the print just echos to the screen, so, what is printed to the screen is also written to the text file.
a print statement automatically indexes to a new line when you hit enter, the f.write statements, the \n adds a new line, otherwise, it just concatenates one long line.

R_

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

SWCASWCA — Sat, 11 Dec 2021 09:54:09 GMT

Again, many thanks for the code! Is there a particular place I have to put the write to text file code? I imagine the f.write line has to go after the print statement but what about all the stuff at the beginning?

So I think it's working, but I did forget that I am not keeping any of the fields once I've converted to a point. I do want to capture the original path, and also a small portion of the path, and calc them to two attribute fields in the appendFC.

So for example, for the following paths:

D:\Projects\12345_XXX\SHP\APE.shp
D:\Projects\34567_XXX\GDB\MyGDB.gdb\SAB

I would want two records in my appendFC where the ProjNum field has been calc'd to be 12345 and 34567 respectively. I'm trying to capture it with this code but it isn't working.

import arcpy, os

#Set the workspace
in_workspace = r"D:\Projects"
appendFC = r"C:\TEMP\PDX_SAB\PDX_SAB.gdb\PDX_SAB_Centerpoint"
# make an in memory dataset its faster.  no need to write to disk if you are just deleting it.
outFC = "in_memory\\centerPoints"

# create list of files from directories and subdirectories
for dirpath, dirnames, filenames in arcpy.da.Walk(in_workspace, datatype="FeatureClass",type="Polygon"):
    # disregard any folder named "ARCHIVE" in creating list
    if "ARCHIVE" in dirnames:
        dirnames.remove('ARCHIVE')

    for filename in filenames:
        if filename.startswith("APE") or filename.startswith("SAB"):
            print os.path.join(dirpath, filename)
            if arcpy.Exists(outFC):
                arcpy.Delete_management(outFC)

            # convert to points    
            arcpy.FeatureToPoint_management(os.path.join(dirpath, filename), outFC, "INSIDE")

            # add field to capture project number
            arcpy.AddField_management(outFC, "ProjectNum", "TEXT", "", "", 20, "", "NULLABLE", "REQUIRED")
            arcpy.AddField_management(outFC, "OriginalPath", "TEXT", "", "", 255, "", "NULLABLE", "REQUIRED")

            # extract ProjectNum from path and calc fields
            exp = str(os.path.join(dirpath, filename)[9:13])
            
            arcpy.CalculateField_management(outFC, "ProjectNum", os.path.join(dirpath, filename), '"' + exp + '"', "PYTHON")
            arcpy.CalculateField_management(outFC, "OriginalPath", os.path.join(dirpath, filename), "PYTHON")
            
            # delete fields; use ListFields to get a list of field objects
            fieldObjList = arcpy.ListFields(outFC)
 
            # Create an empty list that will be populated with field names        
            fieldNameList = []
 
            # For each field in the object list, add the field name to the
            #  name list.  If the field is required, exclude it, to prevent errors
            for field in fieldObjList:
                if not field.required:
                    fieldNameList.append(field.name)

            # delete field
            arcpy.DeleteField_management(outFC, fieldNameList)

            # append to final FC
            arcpy.Append_management(outFC, appendFC)

I'm getting this error:
Traceback (most recent call last):
File "C:/TEMP/PDX_SAB/SAB_Python.py", line 31, in <module>
arcpy.CalculateField_management(outFC, "ProjectNum", os.path.join(dirpath, filename), '"' + exp + '"', "PYTHON")
File "C:\Program Files (x86)\ArcGIS\Desktop10.1\arcpy\arcpy\management.py", line 3129, in CalculateField
raise e
ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000800: The value is not a member of VB | PYTHON | PYTHON_9.3.
Failed to execute (CalculateField).

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

RhettZufelt — Wed, 31 Jul 2013 23:24:00 GMT

the error is related to your expression. Often I find it is easier to create the expression string as a variable and pass this way.
this is working for me:

import arcpy, os  #Set the workspace in_workspace = r"D:\Projects" appendFC = r"C:\TEMP\PDX_SAB\PDX_SAB.gdb\PDX_SAB_Centerpoint" # make an in memory dataset its faster.  no need to write to disk if you are just deleting it. outFC = "in_memory\\centerPoints"            with open(r'D:/Projects/temp.txt','w',0) as f:       # create list of files from directories and subdirectories     for dirpath, dirnames, filenames in arcpy.da.Walk(in_workspace, datatype="FeatureClass",type="Polygon"):         # disregard any folder named "ARCHIVE" in creating list         if "ARCHIVE" in dirnames:             dirnames.remove('ARCHIVE')          for filename in filenames:             if filename.startswith("APE") or filename.startswith("SAB"):                 print os.path.join(dirpath, filename)                 f.write(os.path.join(dirpath, filename))                 f.write("\n")                 if arcpy.Exists(outFC):                     arcpy.Delete_management(outFC)                  # convert to points                     arcpy.FeatureToPoint_management(os.path.join(dirpath, filename), outFC, "INSIDE")                  # add field to capture project number                 arcpy.AddField_management(outFC, "ProjectNum", "TEXT", "", "", 20, "", "NULLABLE", "REQUIRED")                 arcpy.AddField_management(outFC, "OriginalPath", "TEXT", "", "", 255, "", "NULLABLE", "REQUIRED")                  # extract ProjectNum from path and calc fields                 exp = str(os.path.join(dirpath, filename)[9:13])                 exp2 = str(os.path.join(dirpath, filename)                  arcpy.CalculateField_management(outFC, "ProjectNum","exp", "PYTHON_9.3")                 arcpy.CalculateField_management(outFC, "OriginalPath","exp2" ,"PYTHON_9.3")                  # delete fields; use ListFields to get a list of field objects                 fieldObjList = arcpy.ListFields(outFC)                  # Create an empty list that will be populated with field names                         fieldNameList = []                  # For each field in the object list, add the field name to the                 #  name list.  If the field is required, exclude it, to prevent errors                 for field in fieldObjList:                     if not field.required:                         fieldNameList.append(field.name)                  # delete field                 arcpy.DeleteField_management(outFC, fieldNameList)                  # append to final FC                 arcpy.Append_management(outFC, appendFC)      f.write("done with script \n") f.close

I took the liberty to add the file writes to your script. Basically, after the with open statement, the file is open for writing as "f".
so, anytime between the with open, and the f.close if you do a f.write() statement, it will write it to the text file. f.close has to be at same indentation as the with open.
You now have example of how to write a variable, a new row, and text (with new row).

R_

Also, do you use modelbuider? For this kind of script, my initial approach would be to create a model in arcmap to perform this on just one FC and not worry about iterating.

So, in the model, I would pull in the FeatureToPoint tool and convert to in_memory FC
then I'd pull in two AddField tools and configure to create both the fields
then pull in two calculateField tools and configure both
then would pull in the Append tool and configure to append to the appendFC.

now, run the model and make sure it works. If you get errors, fix them until the model runs correctly.
Now, File menu, export, export to python script.

Now you have a starting point, with the proper syntax for most the tools and all you have to do is add the da.walk iterator stuff to make it work on multiples.

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

SWCASWCA — Thu, 01 Aug 2013 16:22:01 GMT

You are right - I went about this sort of backwards. I'm trying to learn more Python and thought the iteration was the place to start (just finding and 'gathering' all the data I needed to work with before doing anything with it). Your suggestion to start with model builder is a good one.

Thanks for the write to text code too! That is all new to me.

Can you tell me why you switched from "PYTHON" to "PYTHON_9.3" in the Calc Field expression? Is that so you can use "exp" without having to escape the quotation marks? From reading other posts I was under the impression that because exp and exp2 were strings they couldn't be used as variables without formatting them as '"' + exp + '"'.

Even using model builder I still can't get the expression right to calc the path to a field. I tried the parse path but either I'm not using it right or that is not it's intention.

Back in the python script I have a feeling the script is not running in the order I specified. I keep getting an error message about the schemas not matching to do the append. When I try printing the fields after the delete statement I see all the original fields - meaning no new fields were added and calc'd, and the old fields were not deleted.

How do I ensure that the processes run in the order I've put them in the script?

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

RhettZufelt — Sat, 11 Dec 2021 09:54:12 GMT

Can you tell me why you switched from "PYTHON" to "PYTHON_9.3" in the Calc Field expression? Is that so you can use "exp" without having to escape the quotation marks? From reading other posts I was under the impression that because exp and exp2 were strings they couldn't be used as variables without formatting them as '"' + exp + '"'.

Because it seems that is the one I can ususally get to work as expected. Guess I'm getting used to haveing objects returned rather than strings.

â?¢To calculate strings to text or character fields, in the dialog box the string must be double-quoted ("string"), or in scripting, the double-quoted string must also be encapsulated in single quotes ('"string"').

Otherwise, just encapulated in a single set of double quotes passes the variable. Not sure where the + exp + comes from.

Basically, the expression needs the quote around it when it gets passed to calcfield. If you didn't put the double quotes around exp2 in the calcfiled line, you would have to include them in the exp2 string, and since that is a calculation, would have to concatenate them on there. Easier to just pass them around the exp variable.

exp2 = '"' + str(os.path.join(dirpath, filename)) + '"'

If you used that for your exp2 variable, it would include the double quotes, so in the calcfield you would just put exp2 without the double quotes.
Hope this makes sense.

Even using model builder I still can't get the expression right to calc the path to a field. I tried the parse path but either I'm not using it right or that is not it's intention.

so, what are you getting for the print os.path.join(dirpath, filename), or what is getting written to the text file for this value? The way I have it, it will return my path u'd:\\working.gdb\dataset' which I wrap in a str() for my exp2 variable. Oh heck, now I see an error in my code that could be causing this (breaking it for sure).

Change this:
exp2 = str(os.path.join(dirpath, filename)

to this:
exp2 = str(os.path.join(dirpath, filename))

After that, see if it works, if not, let me know what the print statements say. Actually, the one that matters is, what does

print exp2 report?

Also, noticed you have set the field lenth to 255, are you sure your path(s) are not exceeding that length? In ArcMap, it will warn you and truncate, not sure how it handles that in code.

Back in the python script I have a feeling the script is not running in the order I specified. I keep getting an error message about the schemas not matching to do the append. When I try printing the fields after the delete statement I see all the original fields - meaning no new fields were added and calc'd, and the old fields were not deleted.

How do I ensure that the processes run in the order I've put them in the script?

Unless you are redirecting it somewhere (sending it to a def()), it will run line by line, so the order is the same. You are not redirecting, so it will do line by line, except, of course where you loop through using for statements.

I suspect that you are running into this issue for append:

Specifies if the schema (field definitions) of the input datasets must match the schema of the target dataset in order for data to be appended.

â?¢TEST â??Input dataset schema (field definitions) must match the schema of the target dataset. An error will be returned if the schemas do not match.
â?¢NO_TEST â??Input dataset schema (field definitions) do not have to match that of the target dataset. Any fields from the input datasets that do not match the fields of the target dataset will not be mapped to the target dataset unless the mapping is explicitly set in the Field Map control.

First, have you ensured that there is an existing point FC named C:\TEMP\PDX_SAB\PDX_SAB.gdb\PDX_SAB_Centerpoint that has two text fields (and is in writeable workspace, and not being viewed in ArcMap or Catalog by anyone (otherwise, lock file..)).

One named ProjectNum with size 20
and
one named OriginalPath with size 255

And then try this for you append line

arcpy.Append_management(outFC, appendFC, "NO_TEST", "", "")

see if that gets past the schema issues. With the NO_TEST, it should copy the attributes that match exactly.

R_

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

SWCASWCA — Sat, 11 Dec 2021 09:54:15 GMT

Well, I ended up figuring it out and as is usually the case with code (that I've found) it all comes down to syntax: quotes, double quotes, = vs ==, etc. I gave up using model builder, as I couldn't figure out the error messages I was getting. I found the errors in the python script to be much easier to work through.

Thanks again for all your invaluable help! Now on to tackling why it creates multiple points each time . . .

Final code:

import arcpy, os

#Set the workspace
in_workspace = r"C:\TEMP\PDX_SAB"
appendFC = r"C:\TEMP\PDX_SAB\PDX_SAB.gdb\PDX_SAB_Centerpoint"
# make an in memory dataset its faster.  no need to write to disk if you are just deleting it.
outFC = "in_memory\\centerPoints"

with open(r'C:\TEMP\PDX_SAB\paths.txt','w',0) as f:

    # create list of files from directories and subdirectories
    for dirpath, dirnames, filenames in arcpy.da.Walk(in_workspace, datatype="FeatureClass",type="Polygon"):
        # disregard any folder named "ARCHIVE" in creating list
        if "ARCHIVE" in dirnames:
            dirnames.remove('ARCHIVE')

        for filename in filenames:
            if filename.startswith("APE") or filename.startswith("SAB"):
            #if filename.name == "APE" or filename.name == "SAB":
                #print os.path.join(dirpath, filename)
                f.write(os.path.join(dirpath, filename))
                f.write("\n")
                if arcpy.Exists(outFC):
                    arcpy.Delete_management(outFC)

                # convert to points    
                arcpy.FeatureToPoint_management(os.path.join(dirpath, filename), outFC, "INSIDE")

                # add field to capture project number
                arcpy.AddField_management(outFC, "ProjectNum", "TEXT", "", "", 20, "", "NULLABLE", "REQUIRED")
                arcpy.AddField_management(outFC, "OriginalPath", "TEXT", "", "", 255, "", "NULLABLE", "REQUIRED")
                arcpy.AddField_management(outFC, "ServiceArea", "TEXT", "", "", 75, "", "NULLABLE", "REQUIRED")

                # set expressions to calculate fields
                exp = str(os.path.join(dirpath, filename)[16:21])
                exp2 = str(os.path.join(dirpath, filename))
                if filename.startswith("APE"):
                    exp3 = "Cultural"
                else:
                    exp3 = "Natural"
            
                arcpy.CalculateField_management(outFC, "ProjectNum", "exp", "PYTHON_9.3")
                arcpy.CalculateField_management(outFC, "OriginalPath", "exp2", "PYTHON_9.3")
                arcpy.CalculateField_management(outFC, "ServiceArea", "exp3", "PYTHON_9.3")
            
                # delete fields; use ListFields to get a list of field objects
                fieldObjList = arcpy.ListFields(outFC)
 
                # Create an empty list that will be populated with field names        
                fieldNameList = []
 
                # For each field in the object list, add the field name to the
                # name list.  If the field is required, exclude it, to prevent errors
                for field in fieldObjList:
                    if not field.required:
                    #if field.name != "ProjectNum" and field.name != "OriginalPath":
                        fieldNameList.append(field.name)

                # delete field
                arcpy.DeleteField_management(outFC, fieldNameList)

                # append to final FC
                arcpy.Append_management(outFC, appendFC)
    f.write("done with script \n")
f.close

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

RhettZufelt — Thu, 01 Aug 2013 19:25:11 GMT

So, what are your polygon datasets in "C:\TEMP\PDX_SAB" ?

You are not doing a selection or anything, so it should make a point for each polygon feature from each input dataset. If you have more than one polygon in the FC, you will get more than one point.

Is this what is going on, or are you getting the same point entered multiple times?

R_

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

SWCASWCA — Thu, 01 Aug 2013 19:41:30 GMT

Yep, you nailed it - multiple records of the same shape (I think someone used it for data driven pages). Fixed with a delete_identical.

I would love to know if there are better/faster ways of doing what I'm doing - either cleaner code or better functions/processes. If this is iterating through a very large workspace with hundreds of folders and subdirectories I can see it really bogging a machine down. TThe in_memory space for example - I never would have thought of it (had never heard of it) and it's really a nice lightweight approach.

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

RhettZufelt — Thu, 01 Aug 2013 21:06:10 GMT

Well, your script is stand alone, and not controlling a map document, so don't think the data driven pages has anything to do with it. you are working with the FC's directly.

From what you have told me, perhaps each polygon feature class has duplicate polygons? In the attribute table of the polygon FC, in ArcMap, how many records do you see? each one would be made to a point. If you are getting identical points, probably identical polygons.

This is real common if the data at one time came from autoCAD.

As far as a different approach, not sure if it would be faster, but here is one method.

Create a list of all the input polygon FC's, iterate them and add your fields to them directly, calculate the fields to thier respective source.

Then, since you already have the list, use that as the input to

arcpy.Merge_management(fcList, outFC) # not sure if you would need to deal with field mappings. I think it will keep all fields that have identical schema.

Then, run the FeatureToPoint on the outFC and do it all in one swoop.

Like I said, not sure if it would speed it up or not, but I have noticed that the append tool is very slow, and involking it each loop could be painfully slow.

R_

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

SWCASWCA — Thu, 01 Aug 2013 21:54:20 GMT

Sorry to be confusing in my earlier post. I meant that the original poly had multiple records in it that someone had used for data driven pages (my guess) and that was why I was getting multiple identical points. The delete_identical worked great for removing those.

I would likely have to deal with different schemas, so it may make sense to add and delete fields one by one before appending, though it sounds like the time hang up may be in the append itself?

I will also at some point have to address the issue that the original polys are in different projections as well but . . . baby steps. 🙂

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

RhettZufelt — Sat, 11 Dec 2021 09:54:18 GMT

OIC. I thought you were running the deleteIdenticals on the final appendFC, but sounds like you used it to "fix" the polygons.

Don't think the different schemas would matter as long as the fields you are concerned about are identical. I.e., if you add your three fields to each polygon FC first.

Now that I think about it (even more), this is what I would try:

merge will add them all together and add all fields from all input FC's. Will combine them if they are identical. So, if they have the three identical fields that you are interested in, the rows will be populated with the attributes.

So,


appendFC = r"C:\TEMP\PDX_SAB\PDX_SAB.gdb\PDX_SAB_Centerpoint"

outFC = "in_memory\\centerPoints"

tmpFC= "in_memory\\tmpFC"

fieldList = ["ProjectNum",  "OriginalPath",  "ServiceArea"]
 
                           ##Merge tool will honor the coordinate system of the first FC in the list to be merged, so I'd make my list something like:
fcList = ["baseFC"]   # where BaseFC is a FC in your workspace that is in the correct coordinate system, this will set the output to the SR you want

layerList = []

then, after you do your walk,
for all file in filenames:
   fcList.append("file")

for all fcs in fcList:                     ## these two look like duplication, but this is how I add to a list with a value already.  For file in filenames is a list I can't control this way
   layername = str(fcs) + "feature_layer"
   layerList.append(layername)    # append the new in memory layer names to a list so we can utilize it after iteration.

   arcpy.MakeFeatureLayer_management(fcs,layername)   # this will put them as in memory layers and are fast

   arcpy.AddField_management(layername, "ProjectNum", "TEXT", "", "", 20, "", "NULLABLE", "REQUIRED")
   arcpy.AddField_management(layername, "OriginalPath", "TEXT", "", "", 255, "", "NULLABLE", "REQUIRED")
   arcpy.AddField_management(layername, "ServiceArea", "TEXT", "", "", 75, "", "NULLABLE", "REQUIRED")

   arcpy.CalculateField_management(layername, "ProjectNum", "exp", "PYTHON_9.3")
   arcpy.CalculateField_management(layername, "OriginalPath", "exp2", "PYTHON_9.3")
   arcpy.CalculateField_management(layername, "ServiceArea", "exp3", "PYTHON_9.3")


Now, after iterating through all FC, and making the list of feature layers, combine them all so we can do the rest in one swoop.
   
arcpy.Merge_management(layerList, tmpFC)   ### merges all the polygon in memory fc's with new calculated fields into an in memory fc

fields = arcpy.ListFields(tmpFC)

for field in fields:
   if field not in fieldList:
          arcpy.DeleteField_management (tmpFC, field)  ## will drop all the "extra" fields if not in the fieldsList.

arcpy.FeatureToPoint_management(tmpFC, appendFC, "INSIDE")   ## this will make point FC in same SR as first input with your three fields appended.

unless you run out of memory or something, I think this approach would be much faster.
Of course, this is just the basic outline, you would need to incorporate the expression code and such in there, so this code isn't "run ready"

Also, as far as converting the SR of the polygons, if you have the need to convert them for some other reason, then probably worth doing that first. however, both append and merge will honor coordinate systems (if defined) and will on the fly re-project them to the ouput datasets SR (to the SR of first input for Merge).

R_

also, could easily skip the two list builders if you make sure that the first one being read to your list is in the proper SR.

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

SWCASWCA — Mon, 05 Aug 2013 16:14:15 GMT

Didn't know about the reproject-on-the-fly with the append and merge commands - good info.

Am trying this but am a bit confused about something - is the for fcs in fcList nested within for file in filenames? Or are they distinct for loops? If they are distinct, how can I rewrite the first two expressions used in the CalculateField command? Right now I have the for loops separated/distinct but then I get an error that 'filename' is not defined when setting up exp and exp2, obviously because it no longer exists outside of it's for loop. Do I use the items in fcList somehow instead?

Thanks

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

RhettZufelt — Sat, 11 Dec 2021 09:54:21 GMT

Melissa,

The for file in filenames just takes the output in filenames (from da.walk) and appends them into a new list.

The for fcs in fcList then iterates through this new list, so they are completely independant of one another. I did it this way so I could control the order of the list and ensure that baseFC is ALWAYS the first in the list so that merge pulls the spatial reference from that layer. If there were an easy way to just add that FC to the beginning of the original filenames list, would just use the one loop instead.

Instead of using filename in your exp statement(s), use fcs.

exp2 = str(os.path.join(dirpath, fcs))

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

RhettZufelt — Sat, 11 Dec 2021 09:54:24 GMT

Melissa,

Was in a hurry when I posted that, and wasn't sure if the filenames list from da.walk is a "normal" python list, so wasn't sure if insert would work correctly. But, after testing, I see that it does.

Could have done it this way:

baseFC = r"c:\pathto\dataset  ##    set to FC that has the desired output SR


                             ## then, after you do your da.walk,
filenames.insert(0,baseFC)    ##  this would ensure the baseFC is the first in the list when passing it to the merge function

Then, you could use the original code (for file in filenames: ) rather than having to use the (for fcs in fcList)

Either way works, but this method is more "clean",

R_

Re: List features in multiple workspaces then convert to point, BUT ignore some paths

SWCASWCA — Tue, 27 Aug 2013 16:09:29 GMT

Just getting back to this after being out of town.

I am now getting a runtime error on the DeleteField_management(tmpFC, field) statement and I'm not sure why. It doesn't give me any clue other than 'error in executing tool'.

I think I am confused by when 'layername' is used and when 'layerList' is used. The layername are appended to layerList early in the script, but then layername is used for adding and calculating fields. Yet, layerList is then used for deleting fields?

Again, thanks for your patience.