topic Re: Copy Shapefiles to Enterprise Geodatabase Feature Dataset in Python Questions

Copy Shapefiles to Enterprise Geodatabase Feature Dataset

JaredPilbeam2 — Sat, 11 Dec 2021 14:36:25 GMT

Is there a way I can copy shapefiles in a directory to an Enterprise Geodatabase Feature Dataset in a stand-alone script? As a monthly update, the shapefiles would replace/update feature classes with the same names in a feature dataset.

I've attempted to copy a shapefile using an SDE connection file with arcpy.CopyFeatures. The script ran but nothing happened.

import arcpy
from arcpy import env
import os

ws = env.workspace = r'path\to\34512.shp'
out_ws = r'path\to\gissql(2).sde'

fcList = arcpy.ListFeatureClasses(ws)

for shapefile in fcList:
    out_featureclass = os.path.join(out_ws, os.path.splitext(shapefile)[0])
    arcpy.CopyFeatures_management(shapefile, out_featureclass)‍‍‍‍‍‍‍‍‍‍‍‍

Re: Copy Shapefiles to Enterprise Geodatabase Feature Dataset

DanPatterson_Retired — Wed, 31 Jul 2019 21:39:00 GMT

https://pro.arcgis.com/en/pro-app/arcpy/classes/env.htm

env.overwriteOutput = True

but Copy Features doesn't seem to deal with so perhaps that is why it doesn't report an error

Re: Copy Shapefiles to Enterprise Geodatabase Feature Dataset

JoeBorgione — Wed, 31 Jul 2019 23:15:29 GMT

As Dan suggests, the env.overwriteOutput = True is not a surefire way. I have found it to sometimes work and sometimes not, with no real pattern.

You could create a list of the shapefile names (sans the .shp extension) and a list of feature classes: loop through the list of feature classes and if the feature class name is in the shapefile list, delete the the feature class.

Re: Copy Shapefiles to Enterprise Geodatabase Feature Dataset

LanceCole — Sat, 11 Dec 2021 14:36:28 GMT

Jared‌,

Take a look at what your statement below is returning. I bet it is a empty set [].

fcList = arcpy.ListFeatureClasses(ws)

Try using the variation below and see if you get a list of shapefiles returned from the folder path.

fcList = arcpy.ListFeatureClasses()

Also, make sure you are just pointing to your shapefile folder path and not a specific shapefile.

ws = env.workspace = r'path\to\shapefiles'

Re: Copy Shapefiles to Enterprise Geodatabase Feature Dataset

JaredPilbeam2 — Sat, 11 Dec 2021 14:36:31 GMT

Joe,

I attempted to put together what I thought you were saying more or less. So far, I haven't got it to work. I put the error below the code. I thought I'd try deleting the feature classes in the dataset first, then copy the shapefiles into that dataset.

import arcpy
from arcpy import env
import os

#directory of shapefiles
shps = r'pathto\Test\shapefiles' 
#directory of FCs in dataset
dataset = r'pathto\TestFGDB.gdb\dFIRM_2019_Effective' 

#create lists for shapefiles and FCs
shpList = arcpy.ListFeatureClasses()
fcList = arcpy.ListFeatureClasses()

#iterate FC list, delete FCs, copy shapefiles to dataset
for fc in fcList:
    arcpy.Delete_management(dataset, "FEATURECLASS") #(in_data, data_type)
    arcpy.CopyFeatures_management(shpList, dataset) #(in_features, out_feature_class)
    print("done")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

line 15, in <module> 
    for fc in fcList:
TypeError: 'NoneType' object is not iterable‍‍‍‍‍‍‍‍‍

Re: Copy Shapefiles to Enterprise Geodatabase Feature Dataset

JoeBorgione — Sat, 11 Dec 2021 14:36:33 GMT

Look at what Lance Cole‌ provided earlier. You need to change your workspace each time you use any arcpy.List##() methods.

A .shp is not considered a feature class but it's a file instead; use arcpy.ListFiles() instead and use '*.shp' as your wildcard. Something like this should work, although I haven't tested it:

arcpy.env.workspace = r'C:\path\to\shapefiles'

shapefileList = []
for s in arcpy.ListFiles('*.shp):
  shapefileList.append(s)


fcList = []
arcpy.env.workspace = r'C:\another\path\to\target.gdb'
for f in arcpy.ListFeatureClasses():
    fcList.append(f)

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

your shapefileList will look like this:

['a.shp','b.shp','c.shp']‍‍‍

While your fcList will look like this:

['a', 'b', 'c']‍‍‍

You will then compare the contents of the two lists. You might get away with something like this:

arcpy.env.workspace = r'C:\path\to\shapefiles'

shapefileList = []
for s in arcpy.ListFiles('*.shp):
  shapefileList.append(s)


fcList = []
arcpy.env.workspace = r'C:\another\path\to\target.gdb'
for f in arcpy.ListFeatureClasses():
    fcList.append(f)

#### notice what workspace we are in ^
### you need to get rid of the .shp in each of the elements in sList
### use list comprehension for that:

newList = [s.replace('.shp', '') for s in slist]

for f in fList:
    if f in newList:
      arcpy.DeleteManagement(f)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

That's what I meant by sans .shp.....

Re: Copy Shapefiles to Enterprise Geodatabase Feature Dataset

LanceCole — Sat, 11 Dec 2021 14:36:36 GMT

Jared,

Add the statement env.overwriteOutput = True to your code as Dan Patterson recommended. This should allow feature classes to be overwritten.
You forgot your env.workspace = r'path\to\shapefiles' statement so your ListFeatureClasses() statement is not returning anything or []
Go back to your original iteration of code - you were very close.

Something like this:

import arcpy
from arcpy import env
import os

shp = r'pathto\Test\shapefiles'
gdb = r'pathto\TestFGDB.gdb'
sde = r'path\to\gissql(2).sde' #if using SDE connection string

env.workspace = shp
fcList = arcpy.ListFeatureClasses()

env.workspace = gdb #or use 'sde' for your SDE database
env.overwriteOutput = True
for shapefile in fcList:
    in_shapefile = os.path.join(shp, shapefile)
    out_featureclass = os.path.splitext(shapefile)[0]
    print('Copying %s to %s' %(in_shapefile, out_featureclass))
    arcpy.CopyFeatures_management(in_shapefile, out_featureclass)

print("done")



‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

A note on using a SDE connection, make sure you are using the correct connection string with the appropriate SDE or DBO level to overwrite a feature class or if need delete a feature class. Your normal connection string may not have the appropriate level or permissions.

Re: Copy Shapefiles to Enterprise Geodatabase Feature Dataset

JaredPilbeam2 — Sat, 11 Dec 2021 14:36:39 GMT

Lance,

The CopyFeatures function failed:

ERROR 000733: Output Feature Class: Same as input Input Features
Failed to execute (CopyFeatures).‍‍

There were no errors up to that point. The print statement on your line 17 printed this:

Copying pathto\Test\shapefiles\S_BASE_INDEX.shp to S_BASE_INDEX‍‍

Apparently, it doesn't like that the shapefiles and the feature classes have the same names.

Re: Copy Shapefiles to Enterprise Geodatabase Feature Dataset

LanceCole — Fri, 02 Aug 2019 14:20:51 GMT

Jared,

You need to change the "pathto\Test\shapefiles" to the path of your shapefiles in line 5 of the code, for example r"C:\myshapefiles\toimport". The same for the gdb path in line 6 needs to be set to the path and name of your GDB file, for example r"C:\myGDBs\TestFGDB.gdb".

Did you catch the second env.workspace = gdb after you populate the fcList? This changes the enfironment from the shapefile folder to the GDB.

Also, if you can post the actual code you are currently using that would be helpful.

Re: Copy Shapefiles to Enterprise Geodatabase Feature Dataset

JaredPilbeam2 — Sat, 11 Dec 2021 14:36:42 GMT

Lance Cole‌

Right, those paths are set right in my actual code.

Here are the files I'm testing on. Ultimately, I'm updating a feature dataset in the sql server database using a sde connection file.

Current Code:

"""
update the feature classes in the feature dataset gisedit.DBO.dFIRM_2019_Effective
from the shapefiles in this directory:
\\gisfile\GISsources\Federal\FEMA\DFIRM\EffectiveProducts\SHP\17197C_WillCo_FIRMdb
"""

import arcpy
from arcpy import env
import os

shp = r'\\gisfile\GISstaff\Jared\Test\shapefiles'
gdb = r'\gisfile\GISstaff\Jared\Test\TestFGDB.gdb\dFIRM_2019_Effective'

env.workspace = shp
fcList = arcpy.ListFeatureClasses()

env.workspace = gdb
env.overwriteOutput = True
for shapefile in fcList:
    print(shapefile)
    in_shapefile = os.path.join(shp, shapefile)
    print(in_shapefile)
    out_featureclass = os.path.splitext(shapefile)[0]
    print(out_featureclass, "\n")
    print('Copying %s to %s' %(in_shapefile, out_featureclass))
    arcpy.CopyFeatures_management(in_shapefile, out_featureclass)
##
print("done")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Copy Shapefiles to Enterprise Geodatabase Feature Dataset

LanceCole — Fri, 02 Aug 2019 15:26:50 GMT

Jared,

You have a single "\" at the start of GBD path this should be a double "\\" in line 12 of your code.