Python scripts that update SDE Layers with NWS local storm report data

CarmenDurham — Thu, 23 Aug 2012 15:06:06 GMT

Hello all,

Not sure if this is an SDE question or a python question, but here goes...

I have been using some python code since late 2009 that was written by John Karagiannis when he was with ESRI. I use the set of scripts that updates a SDE feature class with downloaded National Weather Service's local storm report data. It is run every 30 minutes and is utilized in our FLEX Situational Awareness application. The number of records is NOT large - usually under 100 unless there is a lot of bad weather.

Up until our ArcGIS/ArcSDE upgrade to 10 in April, the script would take 10-20 seconds to run. After the upgrade to 10, I altered the python code to use arcpy instead of gp. All seemed to work okay. But I didn't realize that slowly since April, the time it takes for the script to run reached 18 minutes (for 93 records). Plus I realized that it takes a LONG time to access the feature class in Catalog. Plus it is taking a long time to display in Flex.

I had another copy of the feature class in a TEST database (that would still connect quickly in Catalog) and I tried running the script against it. It still took a lot longer than 10 seconds. Plus, when I set it to run every 30 minutes, the time to process started creeping up higher and higher. After a while, it was again taking longer to connect to the layer in Catalog.

Any idea what is happening and why the switch to 10 changed it all??

More background: This script(s) access a url at Iowa State, downloads a shapefile and saves it locally. Then, it accesses a SDE feature class, deletes all the existing records and appends the new records from the shapefile. This is the portion of the module that does the data load - this is where I see the slow down.

(dst contains the string to the SDE feature class that is unversioned)

def load_data(src, dst):
    if arcpy.Exists(src):
        if arcpy.Exists(dst):
            try:
                #delete the features from the sde feature class
                print "before delete"
                arcpy.DeleteFeatures_management(dst)
                print "after delete"
                #append the new features from the shapefile to the sde feature class
                #print "here"
                arcpy.Append_management(src, dst, 'NO_TEST', '')
                print "after Append"
                write_to_log('  Processed ' + src + ' to ' + dst)
            except:
                write_to_log('  Error Processing ' + src + ' to ' + dst)
                write_to_log('  ' + arcpy.GetMessages(2))
        else:
            write_to_log('  SDE feature class ' + dst + ' does not exist')
    else:
        write_to_log('  Shape file ' + src + ' does not exist')

Thanks for any help you can provide. Maybe someone else is using these same scripts out there since they weren't created just for us!

Re: Python scripts that update SDE Layers with NWS local storm report data

ChristinaMcCullough — Tue, 03 Mar 2015 22:02:47 GMT

Where you ever able to solve this mystery. I had used them on 9.3 and 10.0 spectacularly. I changed jobs and did not have to use them until now. I am using the same scripts and am having some difficulty running them on 10.2.2. I have changed arcgisscripting to arcpy in the code. Any additional comments/suggestions are welcome.

topic Re: Python scripts that update SDE Layers with NWS local storm report data in Python Questions

Python scripts that update SDE Layers with NWS local storm report data

Re: Python scripts that update SDE Layers with NWS local storm report data