topic Re: Spatial Join Slow in Spatial Statistics Questions

Spatial Join Slow

JulianKatz-Samuels — Thu, 20 Jun 2013 15:48:25 GMT

Hi,

I'm using arcmap 10.0 SP5 to do a spatial join with a file that has about 61,000 observations, many of which are null, but it is taking an extremely long time (around 1 day). I thought that it would take a shorter period because I've done spatial joins with bigger files that took a shorter amount of time. Is it typical for it to be taking so long? Is there something that I can do to speed it up?

Thanks,
Julian

Re: Spatial Join Slow

Robert_LeClair — Thu, 20 Jun 2013 16:46:47 GMT

Are you using an ArcSDE feature class, file geodatabase feature class or shapefile?

Regards,

Robert LeClair
Esri-Denver

Re: Spatial Join Slow

MichaelVolz — Thu, 20 Jun 2013 16:54:02 GMT

Are you performing this join in ArcMap or are you performing this task in another environment such as python? If you are doing this in python you would need to add an attribute index on the field in the feature class that is used for the join.

Re: Spatial Join Slow

JulianKatz-Samuels — Thu, 20 Jun 2013 17:16:46 GMT

Hi,

I'm using a shapefile. I'm performing the action in ArcMap.

Best,
Julian

Re: Spatial Join Slow

Robert_LeClair — Thu, 20 Jun 2013 18:13:42 GMT

Sounds like it could be related to "#NIM009568 Spatial joins involving nearest and intersect feature searches are slow to execute." As a workaround, break the data up into subsets or convert your data to file geodatabase feature classes and see if that speeds up the spatial join.

Re: Spatial Join Slow

DanPatterson_Retired — Thu, 20 Jun 2013 18:17:03 GMT

why not create a shapefile for those features that aren't null to reduce the number of calculations that need to be performed, then do the join

Re: Spatial Join Slow

JulianKatz-Samuels — Tue, 02 Jul 2013 13:22:36 GMT

Hi,

I changed it to a feature class and deleted the null values and it is still taking an exteremely long time to do the join. There are about 30,000 observations in the file. Could it just be the size of the file? Is there anything else that I should try or think about? Thanks.

Julian

Re: Spatial Join Slow

Robert_LeClair — Tue, 02 Jul 2013 13:48:56 GMT

Julian - could be the file size. I would experiment and break the datasets into smaller groups (i.e. 10,000 for example) and see if the performance is better/worse.

Regards,

Robert LeClair
Esri-Denver

Re: Spatial Join Slow

MichaelVolz — Tue, 02 Jul 2013 14:37:25 GMT

What data are you trying to spatially join (e.g. Are you trying to join point data to polygon data)?

Also, are you using the Spatial Join accessed from the properties of the layer or are you using the ArcToolbox Spatial Join tool?

Re: Spatial Join Slow

JohnTyson — Thu, 29 May 2014 14:37:22 GMT

I ran into the same situation. In doing some testing, the python script doing a spatial join (closest option) took 10.5 hours, ArcCatalog /Arctoolbox Spatial Join (closets option) looks like it will 11.5 hours as will the using the model builder attempt. The time for the ArcMap Properties Spatial join was about 1 hour. The data used was a 950k line data set join by a 730 polygon data set. The line data is stored in a file geodatabase (fgdb) and in the ArcCatalog test the polygon data is stored in the same fgdb. For the Python script and ArcMap test, the data is stored in a shapefile. The resulting data set is stored in the same fgdb as the line data set.

Re: Spatial Join Slow

curtvprice — Sat, 03 Oct 2015 05:47:40 GMT

These datasets aren't that big. I'm wondering if one could do better putting one or both of the datasets in_memory.

Re: Spatial Join Slow

AlesVanek — Sat, 25 Apr 2020 00:10:13 GMT

Hello,

I found this old post and I am having the same problem as described above.

I am on the latest ArcMap 10.8 - yet with 290k points and 20k objects, the spatial join takes about 20 hours - running on Xeon E5 2680 based cloud.

Appreciate any advice

Regards,

Alex

Re: Spatial Join Slow

curtvprice — Sat, 25 Apr 2020 21:34:03 GMT

Shapefile or GDB? GDB feature classes have spatial indexes built in so I would expect it to run faster.

Re: Spatial Join Slow

AlesVanek — Tue, 28 Apr 2020 20:47:35 GMT

GDB actually helped for the 290k case - it did process in less then 10 minutes.

However my larger file with 600k points as GDB joint with the same 20k objects in GDB will still not complete.

Current run time is ~30 hours and it is stalling. ArcMap allocated ~254 MB out of 64 GB of server memory.

Thanks

Re: Spatial Join Slow

curtvprice — Sat, 02 May 2020 22:57:20 GMT

Just some thoughts

I'd try it in Pro, they have been doing a lot of tiling work to support multiprocessing (and distributed ! processing) and this has made large overlays much more robust and scalable. That and Pro does it all in 64 bit which solves a lot of memory management issues.
If your workflow is tied to ArcMap, and you haven't installed background 64 bit processing, I'd try that - then you get ArcMap tools running in 64 bit which also can help with scalability.

Good luck!