topic Re: Python code in Python Questions

CopyFeatures from Excel data source much slower in 10.1 than 10.0

MoniqueDawson — Sat, 11 Dec 2021 16:29:35 GMT

Hi Everyone,

I have updated my ArcGIS from ArcGIS 10.0 to ArcGIS 10.1 and now my python code is SOOOO SLOW. And worked fine in ArcGIC 10.0!! What is the difference between to two??

Python Code:

### Import arcpy module 
import arcpy
# Local variables:
dir = "E:\\Dawson_Monique\\Weather_Grid\\Louis_Excel_2\\Prov_MB\\"
for nodes in range(1,152):
  name = "nodes_gp_" + str(nodes) + ".csv"
  xyfile = dir + name
  eventfile = dir + "layer\\nodes_gp_"+ str(nodes) + "_layer.shp"
  arcpy.MakeXYEventLayer_management(xyfile, "lon", "lat", xyfile, "", "")
  arcpy.CopyFeatures_management(xyfile, eventfile, "", "0", "0", "0")
  print nodes
print "done"

The Copy Feature portion is Slow. Why? Use to take 5 min now all day.

Thanks

Message was edited by: Curtis Price (edited title, formatted code)

Re: Python code

DanPatterson_Retired — Fri, 11 Jul 2014 22:39:13 GMT

If it is truly taking all day, then I wouldn't blame the python code. Try it on another machine if possible.

Re: Python code

JohnBrodnicki — Thu, 28 Aug 2014 11:27:24 GMT

Just a shot in the dark but you may want to make sure the paths to the ArcGIS python install location are correct.

Re: Python code

DavidWynne — Thu, 28 Aug 2014 16:34:25 GMT

I believe the performance issue is likely due to the heavy use of csv files in your code. There is an outstanding issue NIM103059 that would seem to fit your case.

-Dave

Re: Python code

DuncanHornby — Wed, 10 Sep 2014 14:21:01 GMT

Have the input csv files massively changed in size? Or as Dan suggests has something changed with your machine? Are you getting low on disk space, when did you last run a disk clean up or defrag on your system?

Re: Python code

curtvprice — Mon, 29 Sep 2014 01:12:31 GMT

A really good workaround (that, David, I think should be added to the NIM) is to use numpy. Monique Dawson‌ if you are processing hundreds of files should be much faster even after the bug is fixed (and would have been faster in 10.0 too).

from numpy import genfromtxt
my_data = genfromtxt('my_file.csv', delimiter=',')
arcpy.da.NumPyArrayToTable(array, "in_memory/tmp_xytable")
lyrTemp = arcpy.MakeXYEventLayer_management(tmp_xytable, "lon", "lat", xyfile, "", "")
arcpy.CopyFeatures_management(lyrTemp, eventfile, "", "0", "0", "0")

Re: Python code

DavidWynne — Wed, 01 Oct 2014 22:40:10 GMT

Good idea