SearchCursor vs. export to pandas - What is more efficent

Question

Hi,

Background:
I have a analysis that does cross sections of a multi-patch road model  (around 10000+ sections) to get some geometrical statistics. The statistics would be for each different layer/ feature type (road surface, road shoulder, new terrain etc.) and consist of delta height, width etc. As for now, everything works just fine.

however its a bit slow, as it needs 10000 x #layers = approximately 200 000 SearchCursor requests with where_clause.

Note: the layer in question is a singelpoint PointZ layer with around 1M points

Question:

Would it be any benefit to just loading the entire table with point geometries into pandas and do all the where clauses there? Is there any information about the efficiency of the different cursors?

Does anyone have any thoughts on what is most efficient?

AndreasLorentzen · Answer

yeah... i'm aware. Its kind of a can of worms. And i think the easiest for me is to just test out a couple of methods and see which runs the fastest. Was mainly curious if there was any data on the subject and if there was a definitive solution or best practice 🙂

RogerDunnGIS · Answer

You have a challenging road ahead, focused on optimization.  I'm guessing you mean the optimization of time, not space.  They say space is cheap and time is money!  Obviously a lot of variables have to be considered.  For example, if you were to run with Python Pandas, you may get time X.  But if you were to run with arcpy geoprocessing and have indexed the fields you're grouping on, you may get better time Y.  Database A may perform better than database B.  A Python script may work faster at a command line and slower in a Notebook.

All I'm saying is that I am not well-read in the area of GIS algorithm optimization.  It might be a good idea to use Google Scholar to look for white papers, published articles, theses, and dissertations that address these concepts.  For instance, a quick search just now for "binary search in GIS" pointed me to an article named "CudaGIS: report on the design and realization of a massive data parallel GIS on GPUs."  But that's not stuff I read.

AndreasLorentzen · Answer

Good question. I thought about it, however the tools in it self need to do the where clauses, so if its not implemented in C or some other optimized way (parallel etc.). I could test that as well. I think it there should be some sort of information about the efficiency of different analysis? or a is there an best practice?