topic Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure in Python Questions

Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

EduardoAbreu-Freire — Sat, 11 Dec 2021 10:16:36 GMT

Consider a Feature Class (part of an Enterprise GDB content) and two of its fields are "UTM_grid" and the "OID".

Scripting with python 2.7 for ArcGIS 10.3.

"UTM_grid" field was broken down into "UTM_block" + "UTM_sheet" and then using a dictionary and list structure UTM_block-->UTM_sheet-->OID was organized with this syntax:

{ 'UTM_block_1' : { 'UTM_sheet_A' : [ oid_1 , oid_2 ] }, { UTM_sheet_2 : [ oid_3 , oid_4, oid_5 ] } }

We need to create **shapefiles** with features (identified by the object value - oid) based on the follow data structure (OIDs will be organized regarding an UTM/location based structure):

- UTM_block_* will be a folder

- UTM-sheet_* its subfolder.

- List of objects (OIDs) will populate the shapefile.

---

Another point we have to deal with is that the shapefile should only have some fields of the original Feature Class:

We have used fieldmappings and arcpy.FeatureClassToFeatureClass_conversion() to create a non-UTM-organized shapefile from a Feature Class with the fields that we need.

---

Now the problem is how to process the **feature export** to get UTM-organized shapefiles! How can we manage it?

Bellow is what we have now inside a function:

    with arcpy.da.SearchCursor(feature_class, ['UTM_grid , 'OBJECTID']) as cursor:
        dic = dict()
        for row in cursor:
            UTM_grid_value = row[0]
            oid_value = row[1]

            try:
                b_s_split = UTM_grid_value.split('_')     # split into block & sheet

            except Exception, e:
                pass
                print(UTM_grid_value)
                print(e)

            else:
                # create folders & subfolders (blocks & sheets) in a directory
                dic.setdefault( b_s_split[0], {} ).setdefault( b_s_split[1], [] ).append( oid_value )
                dir_path = os.path.dirname("E:\\")     # full path to directory
                dirs = [ [b_s_split[0]] , [b_s_split[1]] ]
              
                for item in itertools.product(*dirs):
                    if not os.path.isdir(os.path.join(dir_path, *item)):
                        os.makedirs(os.path.join(dir_path, *item))

Re: Scripting (Python) Feature Class to Shapefile into a Data Structure

DanPatterson_Retired — Tue, 26 Apr 2016 09:55:15 GMT

I must be missing where the features to export is coming from assuming that creates the necessary folder structure

Re: Scripting (Python) Feature Class to Shapefile into a Data Structure

EduardoAbreu-Freire — Tue, 26 Apr 2016 10:15:13 GMT

Hi Dan Patterson Blake Terhune

Features classes to export are in a SDE geodatabase. We read/list the FCs in that GDB and for each FC we need to process the export of its features by UTM block-sheet into shapefiles.

The FC includes a field which identifies the location (as shown in code).

e.g. B1SD40B --> breaking down into "block" + "sheet" results in "B1" + "SD40B".

Looking at the fields "UTM_grid" and "ObjectID" we check while sheet ( utm_grid[1] ) = "SD40B", read the corresponding OID, and put the Feature into a Shapefile inside the directory (which has the associated sheet name). The same logic could be used to organize sheets inside the block directories.

Re: Scripting (Python) Feature Class to Shapefile into a Data Structure

BlakeTerhune — Wed, 27 Apr 2016 23:59:53 GMT

I think Dan Patterson is asking how you find the feature class to export? In other words, how do we know where to go to export the OIDs for each sheet. Is there something in the UTM grid data that identifies the feature class to export?

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

EduardoAbreu-Freire — Thu, 28 Apr 2016 16:26:38 GMT

My comment was edited to clarify that point, thank you.

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

BlakeTerhune — Thu, 28 Apr 2016 17:27:15 GMT

So you're iterating a list of feature classes, opening the search cursor on each one, and exporting certain ObjectIDs in the whole feature class based on the ObjectIDs listed in the UTM_Grid field of each record? Seems like that would be ripe for duplicates. Could you upload some sample data?

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

EduardoAbreu-Freire — Thu, 28 Apr 2016 18:19:45 GMT

Blake Terhune yes, we want to export certain OIDs from the whole FC, but that selection is based on the "childs" derived from the "parent" UTM_grid. Childs mean block and its sheets.

Am sending you a simplified FC (polygon) we are using for test which already has the attributes we need.

FOL_250K is a short name for FOLHA (meaning sheet) and 250K (is the scale). I posted the question replacing FOL_250K by UTM_grid. So consider the real name as you need.

https://www.dropbox.com/s/7z6m4rgij3lxlyu/gdb2shp_selected_fields.gdb.zip?dl=0

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

BlakeTerhune — Thu, 28 Apr 2016 18:28:26 GMT

Sorry, I still don't quite follow. I don't see a UTM_grid field in the data you uploaded, nor do I see any lists of OIDs. Do you have some other Python function that dissolves the FOL_250K field into OIDs?

It looks like there are four distinct values in FOL_250K:

BLOCO1 - SULD33S - Namibe
BLOCO1 - SULD33T - Chibia
BLOCO1 - SULE33B - Oncócua
BLOCO3 - SULC33U

Should it make output folders and shapefiles like this?

BLOCO1

.. SULD33S

.... Namibe.shp

.. SULD33T

.... Chibia.shp

.. SULD33B

.... Oncócua.shp

BLOCO3

.. SULC33U

.... None.shp

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

EduardoAbreu-Freire — Sat, 11 Dec 2021 10:16:38 GMT

Blake Terhune

When I posted the problem I wrote "UTM_grid" to give it a more readable name 😃

The file/FC contains the real names. FOL_250K is UTM_grid.

FOL_250K shows those labels (e.g. BLOCO1 - SULD33T - Chibia)

but the value is B1_SULD33T. We do not need to dissolve FOL_250K.

with arcpy.da.SearchCursor(gab_und_crt_dic, ['FOL_250K' , 'OBJECTID']) as cursor:       
        for row in cursor:
            fol_250k_value = row[0]
            oid_value = row[1]

fol_250k_value returns a value in the format (B1_SULD33T).

Output structure we need:

.. SULD33S

.... B1_SULD33S_<featureClassName>.shp

.. SULD33T

.... B1_SULD33T_<featureClassName>.shp

.. SULC33U

.... B3_SULC33U_<featureClassName>.shp

(in this case <featureClassName> will be gab_und_crt_dic)

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

BlakeTerhune — Thu, 28 Apr 2016 20:56:56 GMT

So the value in the FOL_250K field of the sample data you uploaded is not correct? It's hard to parse a value when you don't know what the value is supposed to look like.

Are you saying instead of the value being BLOCO1 - SULD33T - Chibia, it should be B1SULD33T? How do you parse that into block and sheet? At the first S?

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

EduardoAbreu-Freire — Thu, 28 Apr 2016 22:12:24 GMT

Blake Terhune

Missed the "_" in the field.

FOL_250K returns the correct value.

If you run the snippet above with that variable/FC, fol_250k_value = B3_SULC33U

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

BlakeTerhune — Sat, 11 Dec 2021 10:16:41 GMT

Is this close? I'm assuming that you can parse based on the index of "S" in FOL_250K. I'm also assuming you want all fields in the feature class exported to the shapefile.

import arcpy
import os

def main():
    # local variables
    root_dir = r"C:\temp\Msg605039"
    gdb = os.path.join(root_dir, "gdb2shp_selected_fields.gdb")
    feature_class = os.path.join(gdb, "gab_und_crt_dic")

    # Get list of distinct UTM values
    utm_field = "FOL_250K"
    sql_prefix = "DISTINCT {}".format(utm_field)
    sql_suffix = None
    distinct_utm = [
        i[0] for i in arcpy.da.SearchCursor(
            feature_class, utm_field, sql_clause=(sql_prefix, sql_suffix)
        )
    ]

    # Create folders and export features to shapefiles for each UTM value
    for utm in distinct_utm:
        delim = "S"
        delim_index = utm.find(delim)
        if delim_index == -1:
            raise Exception("Could not parse {} with {}".format(utm, delim))
        else:
            block = utm[0:delim_index]
            sheet = utm[delim_index:len(utm)]
            sheet_path = os.path.join(root_dir, block, sheet)
            if not os.path.exists(sheet_path):
                os.makedirs(sheet_path)
            arcpy.FeatureClassToFeatureClass_conversion(
                feature_class,  ## in_features
                sheet_path,  ## out_path
                "{}_{}_{}".format(block, sheet, os.path.basename(feature_class)),  ## out_name
                "FOL_250K = '{}'".format(utm)  ## where_clause
            )


if __name__ == "__main__":
    main()

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

BlakeTerhune — Sat, 11 Dec 2021 10:16:44 GMT

Haha, you're killing me! So if it should look like B3_SULC33U and you want to parse on the first underscore, then you can replace that last section with this (untested):

# Create folders and export features to shapefiles for each UTM value
for utm in distinct_utm:
    delim = "_"
    utm_parse = utm.split(delim, 1)
    if len(utm_parse) == 2:
        block, sheet = utm_parse
        sheet_path = os.path.join(root_dir, block, sheet)
        if not os.path.exists(sheet_path):
            os.makedirs(sheet_path)
        arcpy.FeatureClassToFeatureClass_conversion(
            feature_class,  ## in_features
            sheet_path,  ## out_path
            "{}_{}_{}".format(block, sheet, os.path.basename(feature_class)),  ## out_name
            "FOL_250K = '{}'".format(utm)  ## where_clause
        )
    else:
        raise ValueError("Could not parse {} with {}".format(utm, delim))

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

EduardoAbreu-Freire — Thu, 28 Apr 2016 23:48:34 GMT

Blake Terhune

Made 2 adjustments to get what we need.

Line 22

delim = "_"

Line 28

delim_index+1

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

BlakeTerhune — Thu, 28 Apr 2016 23:51:59 GMT

See my edited reply here.

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

EduardoAbreu-Freire — Sat, 11 Dec 2021 10:16:46 GMT

Blake Terhune

That is it thank you

I am glad you could survive to it

--> Another question-opinion related with this geoprocessing.

The real FCs we have in the SDE have more fields other than those in the file attached to this post.

The most complex FC has a relation with a Table. We investigated FC and Table attributes to get those to build the join. Creating a lyr of the FC we processed the join of it with the Table.

Then using fieldmappings we defined the fields we wanted to keep in the final FC (the one attached) as this snippet shows :

ws = "path to workspace"
fieldmappings = arcpy.FieldMappings()
fieldmappings.addTable(os.path.join(ws, gab_und_crt_dic[0]))

for inputfield in fieldmappings.fields:
    if inputfield.name not in ["ID_UND_CRT","st_UND_GEOL","DSCR_UND_CRT","DSCR_UND_CRT_LG","FOL_250K","ID_UND_LITO",
                                  "DSCR_UND_GEOL", "DSCR_UND_LITO", "IDE_CRON_INF", "IDE_CRON_SUP"]:
            fieldmappings.removeFieldMap(fieldmappings.findFieldMapIndex(inputfield.name))
   
    # export FC with selected fields to shp
    arcpy.FeatureClassToFeatureClass_conversion(gab_und_crt_dic[0], gdb2shp_selected_fields.gdb, "gab_und_crt_dic", field_mapping = fieldmappings)

Would you take this approach or another way?

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

BlakeTerhune — Sat, 11 Dec 2021 10:16:49 GMT

From the code you posted, you're not actually joining a table, only limiting the fields that are exported from whatever gab_und_crt_dic[0] is. If that's the case, then your code is fine. An alternative method would be to only add the field maps you want from the start instead of adding everything, then going through each one to remove what doesn't belong.

ws = "path to workspace"
fc_path = os.path.join(ws, gab_und_crt_dic[0])
out_fields = [
    "ID_UND_CRT",
    "st_UND_GEOL",
    "DSCR_UND_CRT",
    "DSCR_UND_CRT_LG",
    "FOL_250K",
    "ID_UND_LITO",
    "DSCR_UND_GEOL",
    "DSCR_UND_LITO",
    "IDE_CRON_INF",
    "IDE_CRON_SUP"
]
# Build field mappings with only desired fields
fieldmappings = arcpy.FieldMappings()
for field in out_fields:
    fm_obj = arcpy.FieldMap()
    fm_obj.addInputField(fc_path, field)
    fieldmappings.addFieldMap(fm_obj)
# Export feature class with field mappings
arcpy.FeatureClassToFeatureClass_conversion(
    bla,
    bla,
    bla,
    bla,
)

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

EduardoAbreu-Freire — Thu, 05 May 2016 17:15:51 GMT

Blake Terhune

Taking the feature class to shapefile processing organized by utm (sheet) location ( folder: block, subfolder: sheet ) in the case we need to filter the exported output, in other words, create only shapefiles of a specific location (e.g. Block 1: "B1"), what would be the way to accomplish it? That filter-variable would be given as input to the script.

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

BlakeTerhune — Fri, 06 May 2016 18:19:28 GMT

Will you only ever filter on block or do you want to also filter on sheet or something else? Will it be a single filter value or a list of filter values?

Re: Scripting (Python) Feature Class content into Shapefiles based on an UTM data structure

EduardoAbreu-Freire — Mon, 09 May 2016 10:45:26 GMT

Blake Terhune We could manage a filter on block introducing an if condition inside the loop "for utm in distinct_utm".

At a block filter level makes sense to use a single value.

Would be interesting contemplating also a filter on sheet. How could we manage this? Thank you