topic Re: Converting ArcGIS Pro table to in-memory takes extremely long time in Python Questions

Converting ArcGIS Pro table to in-memory takes extremely long time

sbrowneotogo — Thu, 23 May 2024 01:01:44 GMT

As part of a workflow in an python toolbox, I need to grab a table created from a layer in ArcGIS Pro. I've tried several methods, but they all take an absurdly long time. For a table with 50,000 records and <10 columns (weighing in at ~10 MB on disk), it takes well over 3 minutes. I can't expect a user to wait >3 minutes for a tool to run, so I'm trying to see if there's a more efficient method. Here's what I've tried:

tmp=arcpy.da.TableToNumPyArray("temp") # 3 mins 22 seconds arcpy.conversion.ExportTable("temp", my_dir) # 3 mins 31 seconds l=[] with arcpy.da.SearchCursor('temp', field_names) as cursor: for row in cursor: l.append(row) # 3 mins 25 seconds

Is there really no way to get a small file in less than 3 minutes?

Re: Converting ArcGIS Pro table to in-memory takes extremely long time

DanPatterson — Thu, 23 May 2024 05:33:31 GMT

Write geoprocessing output to memory—ArcGIS Pro | Documentation

which workspace type "memory" or "in_memory" ?

Your code doesn't specify. Did you compare the two?

Re: Converting ArcGIS Pro table to in-memory takes extremely long time

JoshuaBixby — Thu, 23 May 2024 14:33:44 GMT

You also didn't say your input data source. If the source is a web-GIS layer, then it may have nothing to do with writing the data out but everything to do with reading the data set over the web.

Re: Converting ArcGIS Pro table to in-memory takes extremely long time

sbrowneotogo — Thu, 23 May 2024 15:49:17 GMT

I think this was the issue - I still don't entirely understand where certain things in ArcGIS Pro are actually stored, but when I reference the layer that exists on the map, it takes ~3 minutes, and I'm guessing it's for the reason you've stated, that it is sending it from a server somewhere across the web to my computer (for the record 3 minutes still seems excessive for a 10MB file). When I look at the properties for the layer, it points to a GDB on my computer. Loading that file using the TableToNumPyArray takes less than a second. Looking at the code for the tool that generates the layer, it looks like it's first writing results to the local GDB and then setting the feature layer to reflect the GDB, so I should be fine just grabbing the GDB instead.

Appreciate the insight that put me in the right direction!