shortest distance between points across the boundary

[LeeRonchip]

Dear colleagues,

I have the following problem I cannot solve. I have house transaction data and catchment boundaries that have been spatially joined, so that I can identify, in which catchment area a transaction took place from the attribute table. I'd like to find the closest matching pairs of transactions located on either side of the boundary in the nearest proximity to each other.
To do so, I wanted to calculate Euclidean distances between the transactions on either side of the boundary and identify the nearest ones. I checked all the tools available in ArcGIS, but they don't seem to allow me to do so. There is a tool that allow me to find the nearest "neighbour" though, but I'd like it to be the nearest "neighbour" located on the other side of the boundary.

I'm new to programming but learned a few bits in python over the last couple of days.
Would greatly appreciate your help in guiding me how to go about this issue and if it's plausible at all.

I'm using ArcGIS 10.0.

Many thanks in anticipation,
Best wishes,
Le

[RichardFairhurst · Accepted solution]

Dear colleagues,

I have the following problem I cannot solve. I have house transaction data and catchment boundaries that have been spatially joined, so that I can identify, in which catchment area a transaction took place from the attribute table. I'd like to find the closest matching pairs of transactions located on either side of the boundary in the nearest proximity to each other.
To do so, I wanted to calculate Euclidean distances between the transactions on either side of the boundary and identify the nearest ones. I checked all the tools available in ArcGIS, but they don't seem to allow me to do so. There is a tool that allow me to find the nearest "neighbour" though, but I'd like it to be the nearest "neighbour" located on the other side of the boundary.

I'm new to programming but learned a few bits in python over the last couple of days.
Would greatly appreciate your help in guiding me how to go about this issue and if it's plausible at all.

I'm using ArcGIS 10.0.

Many thanks in anticipation,
Best wishes,
Le

Of course it is possible.  If the number of joins possibilities in the largest catchment basin extent is not too large you can use the Spatial Join tool, Point Distance tool, Make Query Table tool, and Summary Statistics tool to do this.  All outputs must be done in a File Geodatabase, not a shapefile/dbf table.

1.  Perform a Spatial Join of the catchments to themselves by making it both the target and join inputs.  Use the One to Many option to get the list of all catchment combinations that actually touch each other.

2.  In the Spatial Join output select and delete all records where the Target_FID and Join_FID values are the same to remove all instances of a catchment touching itself.  You now have a listing of all pairs of touching catchments without self touches.

3..  Set up the Point Distance tool to make the same point layer both the input features and near features.  Set the search tolerance to a distance that is approximately two times the distance across your largest catchment basin (so that two points at the extreme opposite ends of the two largest adjoining catchment areas could be matched and chosen when no other point pair is closer together).

4.  Add two new fields to the Point Distance output table called INPUT_CATCHMENT and NEAR_CATCHMENT and make them the same field type as your catchement ID field on the original points.

5.  From the Point Distance table, join the INPUT_FID to the ObjectID of the points and calculate the INPUT_CATCHMENT IDs over and then break the join.  Then rejoin the Point_Distance NEAR_FID to the ObjectID of the points and calculate the NEAR_CATCHMENT IDs over and then break the join.

6.  Use the Make Query Table tool to make a table that does an SQL join where the Point_Distance.INPUT_CATCHMENT = Spatial_Join.Target_FID and the Point_Distance.NEAR_CATCHMENT = Spatial_Join.Join_FID.

7.  Export the Make Query Table to make the query a permanent output

8.  Use the Summary Statistics tool on the export of step 7 and make the Point_Distance.INPUT_FID field  the unique case value and get the Minimum summary of the Point_Distance.Distance field.  This will create a summary table containing the shortest distance to the closest point that is in an adjoining catchment area to the Input point's catchment area.

9.  From the Exported Make Query Table of step 7, join the INPUT_FID to the INPUT_FID of the summary output and select all records where ABS(Distance - Min_Distance) < .001 (or some other very small distance just above your feature tolerance).

10. Export these records to have a table that just contains the two point FIDs in different touching catchment areas that are nearest to each other.

11.  Create a NEAR_FID, NEAR_CATCHMENT and NEAR_DISTANCE field in the original points and join the OBJECTID of the points to the exported table on its INPUT_FID field.  Calculate over the NEAR_FID, NEAR_CATCHMENT IDs and Distance to the points.

[RichardFairhurst]

Model Builder Iterators and Python cursors combined with spatial selections on all points that are not in the same catchment using the Near tool can also do this, but done incorrectly it can take a long time to process.  I prefer to over do the join combinations using geoprocesing and weed out what I don't want to isolate what I do want.  All modeling and coding possibilities have to walk through some level of analysis of irrelevant joins that get weeded out by logic in a problem like this.  Nothing gets to the answer without doing some of that.

I normally do geoprocessing, because it is easier for me to conceeptualize and test.  Once I know what I am doing I may do the code approach.  On complex processes like this the development and debugging time using code takes up so much more time than using geoprocessing tools that I only do the code if I will repeat the process often enough that optimizing the code processing time can eventually make up for the time spent developing the script to achieve that maximum performance.

[LeeRonchip]

Thank you for your prompt reply!
The process you explained step by step is exactly what I've been looking for. Thanks!
I' ve been trying to implement it but unfortunately I'm stuck in a place in which I'm supposed to join the point distance output table input_FID with the ObjectID from the original points. The problem is that the geometry calculation field is disabled.
Im using projected coordinate system so I don't know why i cannot proceed.
Any ideas?

Many thanks.
Le

[RichardFairhurst]

Thank you for your prompt reply!
The process you explained step by step is exactly what I've been looking for. Thanks!
I' ve been trying to implement it but unfortunately I'm stuck in a place in which I'm supposed to join the point distance output table input_FID with the ObjectID from the original points. The problem is that the geometry calculation field is disabled.
Im using projected coordinate system so I don't know why i cannot proceed.
Any ideas?

Many thanks.
Le

No step I mentioned involves the Geometry Calculator and you should not need to use it.  All steps only use the Field Calculator to do attribute field data transfers from fields that should already exist.  So you should be able to proceed.

Also only attempt to calculate fields in the target table, never the tables appended by a standard Join.  Anyway, give the specific step number you are having a problem with, since I don't know what step you are referring to.

All distances that need to be transferred come from the Distance field created by the Point Distance tool in its table output.  It does not come from geometry at all (except when the tool ran against the original points).

[LeeRonchip]

Thanks for that. I got around it eventually and was able to calculate it, as I wanted to. It's been a very educative week and learned a few useful bits that should help in future.
What I wanted to ask however is whether there is an alternative way, I could consider when working with a larger dataset? This operation already took a while and I have a quite significantly larger file to be processed.
Perhaps you know how could I set the near tool up, so that it searches for the points located in the nearcatchment only?

Thanks.

[RichardFairhurst]

Thanks for that. I got around it eventually and was able to calculate it, as I wanted to. It's been a very educative week and learned a few useful bits that should help in future.
What I wanted to ask however is whether there is an alternative way, I could consider when working with a larger dataset? This operation already took a while and I have a quite significantly larger file to be processed.
Perhaps you know how could I set the near tool up, so that it searches for the points located in the nearcatchment only?

Thanks.

No, the Near tool cannot be configured to do that.  Only a python cursor routine could perhaps be developed to optimize the time using a cursor/dictionary process to upload one of the features to memory with the coordinates extracted and optimize the search with an attribute condition.

But in essence you would have to write your own Near tool from scratch and surpass the ESRI programmers for optimization.  It can be done in some cases, but in others they have optimized beyond anything I can create.  Either way, you have to be prepared to spend significant development time on it and run the process a lot after that to make up for the time spent to justify doing that.

Have you at least automated the steps I laid out in ModelBuilder?  The first time always takes a lot of time while you manually set up the steps.  Time savings only takes place when you can just run the process without doing that set up.  If a model is exported to a Python script it can run using Windows Scheduler at off work times when it does not matter if it takes hours to run, as long as you don't have to interact with it.

Anyway, ESRI almost never writes geometry tools that respect attributes or attributes tools that respect geometry.  In part, it is because it is relatively easy to optimize one without the other, but very difficult to optimize the combination of the two.

[LeeRonchip]

No I haven't. Have been doing it in a step-by-step mode up until now. Will see how it goes with scripting though, as suggested. Manythanks. You've been very helpful.