Hot spot analysis on ArcGIS 10.1 running for 2 weeks straight with no end in sight

[PouyanMahboubi]

I'm running a hot spot analysis on points (1.5 million of them). The computer has been processing for 2 weeks straight. ArcGIS10.1 gives no indication of how far along the analysis is, or whether it's even working. Has anyone else run into this?

[DanPatterson_Retired]

did you run it on a smaller dataset to do a time test to get an estimate of how long it would take to run?  Your input data set is very large and I would be surprised if it would finish in an appropriate period of time.

[PouyanMahboubi]

Yes, I ran it on 250,000 points and it completed in about half an hour. On the full 1.7 million points, I tried twice. First time, I stopped it after 3 days thinking it had crashed. Started it again and this time left it running. It says it's still running now (2 weeks later)... But no output yet... Does the analysis time requirement increase exponentially?

[DanPatterson_Retired]

yes it does

[PouyanMahboubi]

So at 20 min processing time for 250,000 points and an exponential increase in time required to complete 1.7 million points --> (1,700,000/250,000)^2 = 46 times longer than 20 min which equals 15 hours. Process has been running for about 350 hours... Tried it on two computers with the same outcome. I used a fix distance band (I let the software auto choose the distance). It chose 2500 m on the smaller data set and 700 m on the larger data set. That should reduce the time requirement shouldn't it (i.e. less points to consider at 700 m)? Is it possible that the software has stalled but not crashed and is not reporting this? Any other possibilities?

[PouyanMahboubi]

The patience paid off. 550 hours and it finally came up with an output!

[MarkJanikas]

The patience paid off. 550 hours and it finally came up with an output!

At least it didn't run out of memory!  The Local Stats shouldn't, as they process a Row-Compressed Sparse Weights Structure that does not require any transpose to calculate... so each row is processed independently and subsequently deleted from memory before proceeding to the next. 

In regards to the previous comment about extrapolating the processing time... The number of features has an order constant processing time ONLY if the number of neighbors is fixed... Like using k-nearest neighbors.... if you are using distance w/ the default then you could have wildly different number of neighbors for each feature (It also needs an additional loop to calculate the default... one to calc the distance that assures every feature has at least one neighbor... and then the calc loop using said distance.) 

We have a change request to allow k-nearest neighbors and delaunay on-the-fly in the tool itself, but for the former we would have to add another parameter... for now you have to use a Spatial Weights File (SWM)... which would have a 2GB limit for 32-bit, so you may be out of luck on that end.

Best bet for a faster solve of Hot-Spot Analysis on a dataset that large is to choose a smaller distance band... it will solve exponentially faster AND that statistic actually does not require that each feature has a neighbor because each feature is considered a neighbor of itself.  For Cluster-Outlier, it wouldn't work.

Best,

MJ