topic Space Time Hotspot analysis in Spatial Statistics Questions

Space Time Hotspot analysis

AnthoniaOnyeahialam — Thu, 08 Dec 2011 21:52:44 GMT

hello,
I have malaria data at hospital level for 20 hospitals on a monthly basis over a period of 10 years. I will like to do some hotspots in space and time but I haven't a clue of where to start even with arranging my data because I have the data separately from the georeferenced hospital database. If I succeed in doing the database, where can I start with the hotspot analysis? is there a possibility to do it in space and time with point data? Txs

Re: Space Time Hotspot analysis

origudes — Mon, 19 Dec 2011 07:30:22 GMT

Probably the best example which is addressing your need is the following link:San Francisco Crime Mapping
http://www.arcgis.com/home/item.html?id=06b67414932343868896c44c7ccda2b5

I think that will help.

Cheers Ori

Re: Space Time Hotspot analysis

LaurenScott — Mon, 19 Dec 2011 18:31:06 GMT

Hi,
You have 20 hospitals, but lots of data over time at each hospital, is that correct? I ask because for most statistics it is important to have at least 30 observations in order to trust your results.

We have lots of resources to help you learn more about hot spot analysis in ArcGIS. Please check out the short videos, free web seminars, and tutorials at www.esriurl.com/spatialstats

At present, the most straightforward way to do a space time hot spot analysis is to create time snapshots of your data, run hot spot analysis on each time frame, then present the results via animation or small map multiples (creating a map for each time period and presenting them together). Alternatively, you could create a custom spatial weights matrix file that links features based on both spatial and temporal proximity. This would be a bit of work. At 10.1 (the next release of the ArcGIS software), our Generate Spatial Weights Matrxi tool will allow you to create those space-time relationships very easily and then use the resultant spatial weights matrix file to perform space-time hot spot analysis. If you can get into the beta program for ArcGIS 10.1, you could get access to that functionality now (??)

If you are interested in constructing the space-time spatial weights matrix manually, here is some information about the file format. You would want features that were within the same space window (like at the same or a nearby hospital) AND within the same time window (malaria count values within 3 days of each other, for example) to be assigned a weighting of 1... otherwise the weighting should be 0. Once you create the space-time spatial weights matrix file, you would simply use it when you run the hot spot analysis tool. Please let me know if you need additional information.
http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/Modeling_spatial_relationships/005p00000005000000/ <-- please scroll to the bottom of the document where it says Spatial Weights Matrix File (.swm)

I hope this is helpful,
Please let me know if I can provide additional information.
Lauren

Lauren M Scott, PhD
Esri
Geoprocessing and Analysis, Spatial Statistics

Re: Space Time Hotspot analysis

PhilipGlasner1 — Mon, 11 Aug 2014 19:28:07 GMT

Hi Lauren,

where can I download the Temporal Collect Events tool that you have been using in this video: What's New in Spatial Statistics: Space-Time Cluster Analysis | Esri Video‌?

Thanks

Philip

Re: Space Time Hotspot analysis

LaurenScott — Tue, 12 Aug 2014 23:38:45 GMT

Hi Philip,

I'm attaching a zip file with the sample tool... it is pretty rough and has only been tested with 10.1 sp1. This question comes up often, so please allow me to answer this question more broadly here:

You want to run a space time hot spot analysis on your event data (crime, disease, traffic accidents) where you don't have an attribute/field to analyze. In other words you just want to know where in space and time you have a statistically large number of events. Some people have guessed that they need to use the Collect Events tool. Unfortunately when Collect Events combines points that are near in space, you lose the temporal component of your data (it will combine two points that are near each other even if they have date stamps that are very far apart). What we need here is a tool that aggregates based on space AND time. You want to be able to set a distance threshold (500 meters, for example) and a time threshold (something like 5 days) and have the tool aggregate only those points that are within both those thresholds. The attached sample script will do that.

Alternative approaches if the attached script doesn't work for you:

1) Create a model tool that iterates through time and selects only those features/events that meet your time requirement... run collect events on the selection set... use the Add Field tool to give the result a DATE field and Calc the value to be a date within the time period selected (like if you want to combine events within two days, you would want to calc the new date field to either the first or second day date for each record output from Collect Events). Then merge all the results into a single file... then create the spatial weights matrix for the merged data using Generate Spatial Weight Matrix, then run hot spot analysis.

2) If you know python, you can try to debug the attached to make it work for whatever version of ArcGIS you are using.

3) You can sign up for the ArcGIS 10.3 beta program and use the space-time pattern mining tool in ArcGIS 10.3 ?? The new Emerging Hot Spot Analysis tool allows you to run space-time hot spot analysis on event data.

I hope this helps!

Best wishes,

Lauren

Re: Space Time Hotspot analysis

PhilipGlasner1 — Wed, 13 Aug 2014 09:36:09 GMT

Hi Lauren,

thank you for your prompt reply. In the meanwhile, I already found the tool that was included in the training course ARC3, version 2.0.

(Anyway, how can users get access to the attached zip file on GeoNet?)

I already tested the Emerging Hot Spot Analysis tool and I think it is very useful in space-time crime analysis. Do you have any further reference on how the categories are calculated?

Thank you,

Philip

Re: Space Time Hotspot analysis

origudes — Wed, 13 Aug 2014 09:41:03 GMT

Thanks Philip

I was about to ask the same question...

Ori

Re: Space Time Hotspot analysis

LaurenScott — Wed, 13 Aug 2014 16:04:26 GMT

Well I am still getting familiar with GeoNet too, but this is what I just tried (and it worked for me):

1) Click on the attachment

2) A pop up is displayed and one of the options is to save.

3) You should see a little blue down arrow when you say okay. If you click on it, you can get the zip file.

If that doesn't work for you I will find someone who can tell us a better solution.

Here is some more information about the Emerging Hot Spot Analysis categories:

*************** Category Definitions ********************

Hot spots that are statistically significant for the last time step interval:

New: only the most recent time step interval is hot

Persistent: at least 90% of the time step intervals are hot, with no trend up or down

Intensifying: at least 90% of the time step intervals are hot, and becoming hotter over time

Diminishing: at least 90% of the time step intervals are hot, and becoming less hot over time

Consecutive: an uninterrupted run of hot time step intervals, comprised of less than 90% of all intervals

Sporadic: some of the time step intervals are hot

Oscillating: some of the time step intervals are hot, some are cold

Hot spots that are not statistically significant for the
last time step interval:

Historic: at least 90% of the time step intervals are hot, but the most recent time step interval is not

Cold spots that are statistically significant for the last time step interval:

New: only the most recent time step interval is cold

Persistent: at least 90% of the time step intervals are cold, with no trend up or down

Intensifying: at least 90% of the time step intervals are cold, and becoming colder over time

Diminishing: at least 90% of the time step intervals are cold, and becoming less cold over time

Consecutive: an uninterrupted run of cold time step intervals, comprised of less than 90% of all intervals

Sporadic: some of the time step intervals are cold

Oscillating: some of the time step intervals are cold, some are hot

Cold spots that are not statistically significant for the last time step interval:

Historic:at least 90% of the time step intervals are cold, but the most recent time step interval is not

We won't be able to get it done for the first release, unfortunately, but in a future release you will be able to select the categories you are interested in and also modify how each category is defined (i.e., right now a persistent hot spot is one where 90% of the time step intervals are statistically significant hot spots and there are no statistically significant cold spots... you might want to change it to 80%, for example).

Beta 5 will have better cell size and time interval defaults when you don't provide anything for those parameters, and messages defining categories.

I hope this helps.

Best wishes,

Lauren

Re: Space Time Hotspot analysis

origudes — Wed, 13 Aug 2014 17:30:04 GMT

Thanks Lauren

always your answer are useful and helpful.

btw, I have managed to download the zip file with

the temporal collect event tool, but it did not work.

i guess I will focus in using the emerging hotspot tool for now.

regards Ori

Re: Space Time Hotspot analysis

LaurenScott — Wed, 13 Aug 2014 18:45:36 GMT

Just curious... what version of Desktop ArcGIS are you using? We will be slammed until the 10.3 release is done, but if we can, we'll try to get the temporal collect events sample script working for 10.2.2.

Thanks!

Lauren

Re: Space Time Hotspot analysis

PhilipGlasner1 — Thu, 14 Aug 2014 07:48:49 GMT

I zipped the Temporal Collect Events toolbox from ARC3 training course which works fine with me using 10.2.2.

Dropbox - TemporalCollectEvent.zip‌

philip

Re: Space Time Hotspot analysis

PhilipGlasner1 — Thu, 14 Aug 2014 07:52:44 GMT

Sounds really great Lauren! Thank you for further information about the tool. I was recently in contact with Chris Delaney about tools for application in Predictive Crime Analysis. Are there any concrete plans for tools (in 10.3 or the next future release)?

btw: found the attachment. may be an issue of my browser

thanks,

philip

Re: Space Time Hotspot analysis

origudes — Thu, 14 Aug 2014 09:45:14 GMT

Thanks so much ALuren

I have downloaded and installed the tool box using ArcGIS 10.2, but i always get error message that it can find the scripts...a bit odd.

I can share screen shot if that helps?

Regards Ori

Re: Space Time Hotspot analysis

PhilipGlasner1 — Thu, 14 Aug 2014 09:51:45 GMT

Did you check whether the script tool references the according script? Right-click on the tool itself (not the toolbox), choose Properties and check in the Source tab whether the according Python script is selected.

(You could otherwise use my zip-file where I imported the python script to the script tool)

Philip

Re: Space Time Hotspot analysis

LaurenScott — Thu, 14 Aug 2014 17:31:48 GMT

Great to know that the sample script is working on 10.2.2! It looks like your zip file only has the toolbox though (instead of both the script and the toolbox). If Ori is using your zip file, that is definitely the reason he is getting an error about not finding the script. If Ori is using the zip file I attached, then Phillip's suggestion to make sure the script path is correct is a good one! Here's how: once you navigate to the Temporal Collect Events toolbox, right click on the tool and select "Properties"... then on the "Source" tab, check to make sure the path to the temporalcollectevent.py script file is correct... if it isn't, browse to the .py file to set the correct path.

Ori: if you still have problems, please send the screen shot to my email, LScott@esri.com, and I will see if I can figure out what's what.

Thanks!

Lauren

Re: Space Time Hotspot analysis

LaurenScott — Thu, 14 Aug 2014 17:49:06 GMT

Our new tools for 10.3 are Create Space Time Cube and Emerging Hot Spot Analysis... they are beta in ArcGIS Pro and we are going to do our very, very best to get them into the 10.3 ArcMap release as well. For future releases we are planning to develop additional tools (probably Outlier Analysis next) to work on the cube (netCDF) data structure. Note: the additional tools work is not yet started, so it technically does not fall into the "concrete plans" category... but that's what we are thinking. Also definitely not yet in the "concrete plans" category: we are very interested in predictive event analysis and are starting to investigate.

Thanks for your interest in our work!

Lauren

Re: Space Time Hotspot analysis

origudes — Fri, 15 Aug 2014 02:45:31 GMT

Ok, i fixed the the pathways of the PY files and now it works..Yay

But some parameters are not clear, for example, is distance measured by meters? so if i insert 1000 is that 1000 meter?

Ori

Re: Space Time Hotspot analysis

origudes — Fri, 15 Aug 2014 04:00:42 GMT

I have also noticed that Lauren Bennet has presented the results in 3d view, which i was very cool and intuitive, is there any explanation or steps how to it? (present the spatio temporal results in 3D?)

Thanks O

Re: Space Time Hotspot analysis

TingLee — Fri, 15 Aug 2014 21:18:37 GMT

"But some parameters are not clear, for example, is distance measured by meters? so if i insert 1000 is that 1000 meter?"

For the parameter "Distance Threshold", the distance unit is base on the output coordinate system. By default, the output coordinate system will be the same as the input's projection spatial unit. For example, if the projection of the input is WGS1984 (Web Mercator Auxiliary Sphere), then the distance is measured in meters. If the projection is NAD 1983 StatePlane, then the distance is measured in feet.

Thanks,

Ting-Hwan

Re: Space Time Hotspot analysis

PhilipGlasner1 — Mon, 18 Aug 2014 06:40:29 GMT

Thank you Lauren for the information.

I am doing my PhD in Predictive Crime Analysis and are also working for the local Esri distributor in Austria, so that's why I am always interested to use Esri software in my research. It is good to hear that there are ideas to extend analysis methods in this area.

Philip