Doing a geospatial join in GeoEvent

1041
2
07-09-2019 12:08 PM
DaveClemmer
New Contributor

We have information about ships coming in, and would like to augment the placement information with information about the depth of the water where they are.  We have a feature service with the depth info, in the form of lines of depth.

So I want to do something like find the nearest line, take the depth from it, and apply that depth to the ship's info.

The only way I can come up with, that I'm sure is feasible, is to write a custom processor that will query the depth data,  getting all the lines, check the distance to each line using the java geometry library, and pick the closest one.  I was hoping there was an easier way, both for ease of development (the Java API for querying seems horribly cumbersome, and we're new to this) and for efficiency (any spatial index will be bypassed, and this will get called a lot).

So, is there a way to query the feature service such that the results are sorted by distance from a given point?  Or a way to just get the closest feature, explicitly?  The only distance usage I can see in the query interface is to get all features within a set distance from the point I'm looking for, which is of limited use.  And the closest I can find is the GeoAnalytics REST API has a findNearest, but that doesn't seem terribly helpful, for several different reasons (not least that I'm looking for nearest to one point, not for every point in a feature service.  Also, the limit of 5k features would likely be problematic).

Also, to stave off any suggestions using outside services, I need to point out that this will be running on an isolated network.  So unless that entire service can be downloaded, it won't be helpful.

Thanks in advance for any pointers,

Dave Clemmer

0 Kudos
2 Replies
RJSunderman
Esri Regular Contributor

Hello Dave –

I am not familiar with a GeoEvent Server processor, out-of-the-box, that would allow you to configure a query to a feature service and then sort the returned feature record set to identify the one whose distance from an event record’s point location is the smallest. That sort of functionality would have to be developed as a custom processor using the GeoEvent Server's Java SDK.

The objective, if I understand what you want to do, is to locate the bathymetry line closest to a point (denoted by the blue dot in the below illustration) which represents the point geometry received as part of a tracked asset’s location/status event record. You want to enrich the received event record with a “depth” attribute taken from the polyline feature.

Bathometry - Find Nearest

GeoEvent Server can model a set of geofences using bathymetry polylines, but I would not recommend using a spatial operation such as INTERSECTS or TOUCHES to determine a spatial relationship between a point and a polyline as there is no way to specify a tolerance.

The Esri Java Geometry API used by GeoEvent Server for all spatial relationship tests does not provide a NEAREST spatial relationship operator to allow the determination of the closest bathymetry line to a vessel’s reported position, which makes what you want to do difficult using out-of-the-box capabilities.

For example, out-of-the-box, you can use a Field Enricher processor to enrich a received event record with attribute and/or geometry from a “related” feature record, but that processor only supports an attribute join for event enrichment. GeoEvent Server does support a spatial join, but only through the GeoTagger processor, which limits the event enrichment to incorporating the name of a spatially relevant geofence.

Eric Ironside‌ suggested an approach in which he used a custom processor to obtain a pixel value from an image service. In his case the image service data was elevation contours, but the pixel could just as easily provide the depth for bathymetry. He developed his custom processor to enrich GPS data that wasn’t reporting elevation with with the z-values.

Gregory Christakos‌ suggested that developing a custom processor to invoke a GP Tool, via a REST endpoint as a GP Service might be an option. I'm not sure which GP tool you might want to use, Greg suggested ArcGIS Pro's Near tool. GeoAnalytics includes a similar tool in their standard analysis tool set for batch analysis. There's also the Find Nearest tool in the ArcGIS Enterprise portal’s standard analysis toolset. Perhaps more important than which tool you want to try and expose as a GP Service is how you would obtain the bathymetry geometry to pass to the tool along with your vessel's real-time location.

I suppose you could use GeoEvent Server to buffer the vessel's reported point location by a discrete distance, then use a GeoTagger to get the "names" of bathymetric polylines you had loaded as geofences which intersect that buffer, then Field Enrich to pull the geometries of the polylines into the event record ... but that's a lot of work to do in advance of using a custom processor to invoke an operation, which is where I think you're going to end up. 

Determining the optimal distance to use when buffering the vessel's location, to try and guarantee intersection with at least one bathymetry line without intersecting multiple bathymetry lines could be difficult. I think your concerns regarding event volume, efficiency, and leveraging spatial indexes are also justified. The number of spatial relationship queries a custom processor might need to make to determine the “nearest” bathometry line to a point could devastate overall event throughput. If you want development help from an engineer in Esri Professional Services, let me know and I can ask someone to reach out to you.

– RJ

DaveClemmer
New Contributor

I have been a terrible netizen, not responding to this some time ago.  But I have done a lot of work on it, since, and actually have something working, albeit slowly.  So I have comments on several parts of the reply.

GeoEvent Server can model a set of geofences using bathymetry polylines, but I would not recommend using a spatial operation such as INTERSECTS or TOUCHES to determine a spatial relationship between a point and a polyline as there is no way to specify a tolerance.

A quibble with this, perhaps.  If you do a feature service query, you can specify an input geometry (in my case, a point), along with a distance and distance units, and using INTERSECTS, that will effectively construct a buffer around the input geometry and intersect with that buffer.  Using a geofence directly, I know nothing.  But because our problem already involves some complicated (possible overly so; we haven't yet determined that) geofences, modeling bathy lines as a geofence is likely to be a non-starter.

The Esri Java Geometry API used by GeoEvent Server for all spatial relationship tests does not provide a NEAREST spatial relationship operator to allow the determination of the closest bathymetry line to a vessel’s reported position, which makes what you want to do difficult using out-of-the-box capabilities.

This is true, but it does have a DISTANCE operator.  I can't find anything which documents what units the answer is in (or, similarly, any way to convert them to known units like meters or miles), but it does at least allow for manually determining a nearest answer.

So, what I'm currently doing is basically a combination of those two operations.  I'm constructing a buffer around the point to get all features within a specified distance, then checking the relative distances of those features to get a nominal depth.  As I implied earlier, this works, but is not fast (~0.5s or more).  I'm trying to see if I can do any caching to improve that at least a little bit, but I suspect my limit on that will be a lot less than 50% (and even that might not be fast enough for this to be practical.  I also have not really played with the size of the buffer, which might also significantly help.

Incidentally, in looking at this, I ran into several bugs in DefaultArcGISServerConnection.getAllFeatures(..., Map, ...).  I filed bug reports, and would love to have confirmation that these didn't get tossed away (never tried filing bugs before, so not even sure if I went about it the correct way).  Regardless of those bugs, I found it necessary to do my queries directly in HttpClient (I did not try to use the GeoEventHttpClient).

Eric Ironside suggested an approach in which he used a custom processor to obtain a pixel value from an image service. In his case the image service data was elevation contours, but the pixel could just as easily provide the depth for bathymetry. He developed his custom processor to enrich GPS data that wasn’t reporting elevation with with the z-values.

I would be very curious to know more about this possibility, as I looked, briefly, at Image Services, and didn't see any way to query them.  But since the feature service I'm using to query is based on images converted to geometries, it's possible that this could end up being a very worthwhile approach.

Gregory Christakos suggested that developing a custom processor to invoke a GP Tool, via a REST endpoint as a GP Service might be an option. I'm not sure which GP tool you might want to use, Greg suggested ArcGIS Pro's Near tool. GeoAnalytics includes a similar tool in their standard analysis tool set for batch analysis. There's also the Find Nearest tool in the ArcGIS Enterprise portal’s standard analysis toolset. Perhaps more important than which tool you want to try and expose as a GP Service is how you would obtain the bathymetry geometry to pass to the tool along with your vessel's real-time location.

I did mention, in my original message, why Find Nearest would not work for me.  Find Near (in GeoAnalytics, at least) has the same, disqualifying to me, limitations.  I am curious how it would be possible to use any ArcGIS Pro GP tool, even though I must admit to some reservations about how feasible to use they would be.

Determining the optimal distance to use when buffering the vessel's location, to try and guarantee intersection with at least one bathymetry line without intersecting multiple bathymetry lines could be difficult. I think your concerns regarding event volume, efficiency, and leveraging spatial indexes are also justified. The number of spatial relationship queries a custom processor might need to make to determine the “nearest” bathometry line to a point could devastate overall event throughput. If you want development help from an engineer in Esri Professional Services, let me know and I can ask someone to reach out to you.

I wouldn't worry much about getting the response down to only one line immediately from the query, but (thinking out loud, here, as it were) it would be cool if we could use a GP tool to query our bathy feature set to find an optimal buffer size.  I'm thinking that if we could do a Find Nearest from each bathy line, and get the furthest distance between bathy lines, then doing a buffer of a hair more than half that distance would probably be ideal.  Perhaps we would even have a different answer for each layer... (We're also a bit limited for the moment.  We plan on adding one or more GA server(s) at some point, but that's at an indefinite point in the future.  But if this is something that can be run now from Pro, that might get us going.  _I_ can't currently run Pro, but we do have at least a couple of installs here.)

I'll have to look into what's involved in getting someone here for development help; I'm not sure if that's feasible, but it would definitely be nice.

Thanks for the response,

Dave Clemmer

0 Kudos