topic Re: Spearman's rank or Pearson's correlation coefficient? in ArcGIS GeoStatistical Analyst Questions

Spearman's rank or Pearson's correlation coefficient?

CharlotteJ — Sun, 25 Oct 2015 09:47:53 GMT

This isn't a GIS question so much but I'd like to correlate the percentage of the population of census output areas who are children with mean distances to green spaces, but am unsure whether to use Spearman's Rank or the Pearson's Correlation Coefficient. I understand that Spearman's rank is best used for ordinal variables, which I don't think either of these 2 are, so perhaps it's better to use Pearson's? But then I've also read that Pearson's is better used when the relationship between variables is linear which I'm not sure it is in this case, so I'm unsure which option's better. Any advice would be much appreciated.

Re: Spearman's rank or Pearson's correlation coefficient?

DanPatterson_Retired — Sun, 25 Oct 2015 12:58:15 GMT

why not categorize your variables into classes, the use a chi-square test of difference/association between the two variables. This would allow you to see whether areas with high percentages of children live in areas where the walk distance to green space is low. You aren't implying causality in this type of response, since I am sure that is not what is intended.

Re: Spearman's rank or Pearson's correlation coefficient?

CharlotteJ — Sun, 25 Oct 2015 21:25:23 GMT

Thank you for your response. No I'm not intending to imply causality, I'd just like to investigate whether areas with higher percentages of children tend to have poorer or better access to green spaces, so yes, maybe chi squared would be better then. I'm completely new to this though because I haven't done a chi squared test before so I'm a bit uncertain about trying to do one since this is for my dissertation. When you suggest categorizing the variables into classes do you mean say dividing the percentages and distances into categories like for example 0-20%, 20-30% etc and 0-100 metres, 100-200 metres etc. Then would I cross tabulate these in a table to fill with counts of the number of output areas which fall within each category? Then use this to perform the chi squared analysis?

Re: Spearman's rank or Pearson's correlation coefficient?

DanPatterson_Retired — Sun, 25 Oct 2015 22:31:59 GMT

Yes as you describe. It may be useful to use existing classifications if they are available in the census categories, or you could use means and std devs to produce your classes or just subdivide the % data based upon some criteria of your choosing.

for example

high > =1 std dev

norm -1 std - 1 std dev

low < = -1 std

as an example should the distribution appear normal, this will give you some supportable criteria. Should the %age data not be normal, then you might want to examine the distribution to assess break points.

The key point is .... your categories, when derived from interval/ratio data, need to have some kind of justification! you could get the "how did you produce those classes" or "why did you choose those classes" questions (we examiners are not a cruel lot...just looking for thoughtful consideration and not a "because?!" person)

in any event, plot you data first to see if there is any clustering/pattern/ etc in the data, then do your descriptive statistics...then and only then, choose your inferential test. Parametric statistics has its requirements and if the data don't conform, then your non-parametric options (eg. chi) can step in.

Good luck and post more questions and/or graphs if needed.

PS I hope your advisor and committee are giving you similar advise.... take theirs over mine should there be a disagreement. If none is forthcoming, pose these issues with them...there are individuals that really don't care about appropriateness of the test and would say use pearson's but do it versus transformed data... eg log(%) versus sqrt(distance)+0.137 ... just so the distribution becomes 'normal'-ish but that is another isssue

Re: Spearman's rank or Pearson's correlation coefficient?

CharlotteJ — Mon, 26 Oct 2015 13:17:00 GMT

Ok then, thank you

My supervisor seemed to think that using either pearson's or spearman's would be ok, but wasn't sure which would be more appropriate.

Can I just clarify that both my variables are ratio variables? If they are then I'm assuming that spearman's probably wouldn't be as appropriate to use as pearson's since spearman's is better used for ordinal variables? But I understand that variables have to vary linearly in order for pearson's to be carried out and I'm uncertain whether this is the case or not for mine? Here are the scatter diagrams I've plotted for them at the moment. I'm also measuring access for older individuals and those considered disabled, as well as for children.

Or if I transformed the data so that it becomes normal, as you describe, would it solve the data not being linearly related being an issue? I'm not too sure how to go about doing this though. Thank you.

Re: Spearman's rank or Pearson's correlation coefficient?

DanPatterson_Retired — Mon, 26 Oct 2015 15:10:23 GMT

Well... a quick look at the data would strongly suggest that regardless of the variables and the measurement level, there is no difference/association or correlation between them on face value. You could try grouping the data to see if there are clusters within the scatter plots. You could also try to map the data. Don't forget, that buried within that data may be the fact that some people may not be able to live close to green space regardless how much they may want to use it. A person's place of habitat is not a free variable since it is controlled by availability, cost and whole slew of other factors. Also, green space/ parks aren't able to be located freely in the landscape. So in short, you have two things that may be directly related, HOWEVER, they are unable to meet in the 'spatial middle' so that demand and supply is met.

I wouldn't spend too much time trying to prove anything with the inferential statistics...basically if you need to show, or are asked to show these data, don't spend much if any time desperately grasping at straws (ie the last graph and the outlier where block has 90% of the people over 65 and they live really really close to green space...perhaps there is a park directly across the street purely by coincidence)

Re: Spearman's rank or Pearson's correlation coefficient?

CharlotteJ — Mon, 26 Oct 2015 18:19:19 GMT

Ok then, that makes sense. Thank you. No I'm not surprised that there doesn't seem to be clear relationships for any of the graphs. Would carrying out clustering analysis enable me to see whether there are areas with similar percentages of the social groups with similar access to the spaces?

But perhaps carrying out a statistical analysis won't be worthwhile at all then since the graphs show fairly clearly that there isn't a relationship between any of the variables. Or perhaps I could do one just to clarify that there isn't a relationship. I may just discuss how the graphs show that there are variations in access for both output areas with higher percentages and lower percentages of individuals from each of the groups. And I can discuss how the areas with higher percentages shown to have slightly poorer access are perhaps in the greatest need of improvements to access.

Re: Spearman's rank or Pearson's correlation coefficient?

DanPatterson_Retired — Mon, 26 Oct 2015 19:31:21 GMT

I would definitely produce a map showing the data... green space and some combination of the others. It may reveal more information than is given by the simple data points themselves. Start with something simple so one can see the pattern of where things are, where things can be then get a screen grab or two and post them. From there more ideas may come forth

Re: Spearman's rank or Pearson's correlation coefficient?

CharlotteJ — Mon, 26 Oct 2015 21:42:46 GMT

Yes, I've actually already made some maps Here are the ones I've created to assess variations in access for the older individuals of the population, to identify areas which priority should perhaps be given to in improving accessibility. The first shows the percentages of the population of each output area who are aged 65 and above, as well as the distribution of green spaces. And the second shows the average distances to the closest green space for each output area.

It's a bit hard to compare them both though because of how many output areas there are and how small they are. But I suppose it can just about be seen that the average distance to the closest green space is fairly low for the only area with over 80% of the population over 65, being less than 400 metres. Then most of the areas in which 40-60% of the population are over 65 have slightly higher average distances of between 400-1200 metres. But most areas with poor access, over say 1800m, seem to have smaller older populations. So perhaps it could be said that, in this case, the distribution of the spaces seem to be reasonably fair in the sense that areas where access is poorer, above say 1200m, tend to have smaller populations of older people, if it's assumed that good access should be prioritised for older people. But I can use arcmap to analyse this a bit more carefully though by looking at the attribute tables for each variable.

I should probably change the classification colours for the second map so that it's a bit easier to distinguish between the categories for the output areas as well. And perhaps it would be good to combine both maps into one, but I can imagine this would be quite difficult to interpret and not very clear.

Re: Spearman's rank or Pearson's correlation coefficient?

DanPatterson_Retired — Mon, 26 Oct 2015 22:12:32 GMT

A couple of things.

the classifications schemes are the result of? .... if I change it, the map will change... remember, lie with statistics, lie with maps
dump the decimals
percentage population is one thing...but...what is the absolute population? 6 elderly in an area with 6 is not the same as 6000 in an area with 6000, I will leave you to draw a conclusion
older people are more likely to live in the same area for a longer period of time, hence the infrastructure builtup around them...if proximity to parks is such an issue why didn't they move to greener pastures?
you will have to deal with the mobility issue at some point... ie can people move or locate freely within the landscape
what are the impedances to locating green space? If we are talking parks, then there are areas that should be taken off the map...ie industrial parks, government enclaves, universities (unless these have publically accessible parks)
what is the age distribution of greenspace? Is there any way to find out when parks were first created? I know that provision of greenspace where I live is a subdivision requirement. It is not distance based but driven by other factors.
and on...

Re: Spearman's rank or Pearson's correlation coefficient?

CharlotteJ — Tue, 27 Oct 2015 00:12:03 GMT

Sorry I'm not sure that I understand your first point, I decided on the classifications myself by just dividing up the range covered by each of the variables, so that each separate category covered an equal range. I thought this would be the clearest way of showing how access and the distributions of each group vary across the city. Or would you suggest using natural breaks in the data to determine the classifications?

And yes, dropping the decimals sounds sensible, and possibly using absolute numbers instead of percentages also seems to make sense. Because I suppose that one area may have a lower number of older people than another, but the percentage may be higher. Or perhaps mapping population density may be sensible? Then the area of each of the output areas can also be accounted for, as well as the absolute populations, so may be a better indicator of the distribution of each social group.

And thank you for your other points, I'll take them all on board. I'm intending on writing about limitations and uncertainties such as these in my discussion as I understand that they're all valid points. And regarding your 6th point, I only mapped out spaces I understood were publically accessible.

Re: Spearman's rank or Pearson's correlation coefficient?

DanPatterson_Retired — Tue, 27 Oct 2015 00:45:54 GMT

your classification is fine...albeit qualitative and probably doesn't differ much from NBs ... of course you could make a statement to some effect should you have investigated that (remember....I am playing devil's advocate)

population density would be a reasonable compromise and may yield other information. You may find that total population density may reveal a different pattern with green space itself.

In reference to the 6th point...it may be worth while to remove spaces where people don't live and green space couldn't exist...examples given in my list. This will also affect population density. Consider a 1 km^2 area, 95% by gov't buildings, 2.5% green space and the remaining 2.5% seniors residence....you conclusion would be?

The nice thing about talking about the limitations up front is that you show that some thought went into the whole project...it won't ever be perfect...in 5 years you will think of something else you could have done...in 10 maybe more...but you have to give it up sometime. Your only obstacle right now is to get a thesis done and a defense (if applicable) completed by warding off the external advisor who asks those innocuous questions that you hadn't thought of. The one that threw me was ... and how does you work fit into the bigger picture of (your program here)????

Re: Spearman's rank or Pearson's correlation coefficient?

DanPatterson_Retired — Tue, 27 Oct 2015 00:56:49 GMT

make sure you are aware of the capabilities of the two main toolsets

An overview of the Spatial Statistics toolbox—Help | ArcGIS for Desktop

An overview of the Geostatistical Analyst toolbar and toolbox—Help | ArcGIS for Desktop

so you aren't caught off guard. you can dismiss an approach by putting the analysis in context of the importance to your discussion

Re: Spearman's rank or Pearson's correlation coefficient?

CharlotteJ — Tue, 27 Oct 2015 12:50:14 GMT

Ok then, and yes perhaps I will change the percentages for each output area to population densities then.

Thank you for your advice, yes as long as I show that I've acknowledged any uncertainties surrounding my study, I think this will be ok.

And in terms of where my work fits into the bigger picture of geography, I suppose that measuring accessibility to services is important for comparing the spatial distribution of demand relative to supply, and looking at how access can vary spatially. As access to green spaces is generally considered to be associated with improvements in wellbeing, assessments into the adequacy and equality of access across cities seems important to investigate whether people have equal opportunities to yield these benefits to wellbeing, regardless of where they live. I can then use my assessments to identify areas which are potentially in greatest need of improvements to access. Hope that addresses the question.

Throughout my study I've measured distances to different types of spaces and calculated the percentages of the population for which different accessibility standards set for green spaces have been met. However I feel that these measures are all quite simplistic, since my results only really consist of average distances and percentages. So this is why I was hoping to bring in some statistical analysis by testing to see how the distributions of social groups vary with access, and whether green space are therefore well located relative to demand. But would you suggest that this isn't really worthwhile since there clearly won't be strong relationships between the variables? But if I do decide to use Pearson's to measure correlations between the variables, would you mind explaining what would be involved with transforming my data as you previously mentioned to do? You mentioned to log the percentages, rather than using the original values?

Re: Spearman's rank or Pearson's correlation coefficient?

DanPatterson_Retired — Tue, 27 Oct 2015 13:18:03 GMT

I think your background support is fine

When you talk about distance, you have been using Euclidean distance (ie crow-flies distance) and not network distance (ie travel along roads). This can be mention, but do not do it since it opens up a whole can of worms and will mask anything that you would gain.

As for transformation...I was trying to make the point that people will go to elaborate lengths to use parametric statistics (ie simplistically, for data which has a normal distribution) rather than use there non-parametric equivalent (ie pearson's versus spearman's) because they think that parametric statistics are somehow superior...they aren't. So if you can explain what ... log(%>65) + 0.25 really means then don't go there (the answer by the way is ... the distribution is totally weird and this is the equation that made it look normal. Other things I have heard... my advisor told me to normalize it. Or ... not sure, everyone uses pearson's don't they?!?)

Finally you aren't studying a correlation, you are studying an association ... should an examining board start arguing over the semantics, just let them go at it and keep out of the fray it boils down to causality at times.

Re: Spearman's rank or Pearson's correlation coefficient?

CharlotteJ — Tue, 27 Oct 2015 17:42:30 GMT

Oh good. And I have actually been measuring distances by network distance, using Arcmap's network analyst extension. I see, so do you think it would be fine using spearman's rank over pearson's then? I would do chi squared but I don't feel very confident with it as I've never done it before. But I didn't think that Spearman's should be used for testing the association between 2 ratio variables (distance and percentage), as I was under the impression that at least one variable should be ordinal in order to use it?

Re: Spearman's rank or Pearson's correlation coefficient?

DanPatterson_Retired — Tue, 27 Oct 2015 17:58:39 GMT

Personally, I would like to see you class the data into groups and run the chi square test...in spreadsheets

=CHISQ.TEST there is an example in there. or for spearman's... Dr Google has many links.

http://blog.excelmasterseries.com/2014/05/spearman-correlation-coefficient-in.html

Re: Spearman's rank or Pearson's correlation coefficient?

CharlotteJ — Tue, 27 Oct 2015 18:22:28 GMT

Ok then, I'll give it a go if that would be more appropriate. And thank you for the link. But just for future reference, I've done quite a bit of google searching but am still a bit confused as to whether it is alright to use 2 ratio variables to carry out spearman's rank. Is it ok to do this then, even if one of the variables isn't ordinal?

Re: Spearman's rank or Pearson's correlation coefficient?

DanPatterson_Retired — Tue, 27 Oct 2015 19:06:12 GMT

No..... that is why you have to rank them before pearson's since it is the correlation between the ranks and not the actual data itself... although Chi is the best since it is an association/difference test and not a correlation which implies something else.... trust me...classify your data, get the counts in each class, the expected values are easy to determine then.... it's Chi time!!!

Re: Spearman's rank or Pearson's correlation coefficient?

DanPatterson_Retired — Tue, 27 Oct 2015 19:14:27 GMT

I also wonder why no else is chiming in here? Is everyone else a mathphobic I hope you are running these ideas by your advisor...I want to make sure you are on her/his page and not just listening to me...

Also should you have some time and want some amusement and/or eye openers, you should check out

Academia on Stack Exchange

It often provides me with much merriment and often insight.