Ellipse Calculator Processor workaround in GeoEvent?

2308
7
04-20-2017 12:20 AM
AndersJensen
New Contributor II

I have filed this BUG (BUG-000099124 ) some time ago, and I am wondering if anyone has a good workaround until someone fixes it? Basicly the problem is that Ellipses created as an output when using the Ellipse Calculator Processor have the same size, shape, and angle of rotation when the values of the 'Major Axis Source', 'Minor Axis Source', and 'Rotation Source' properties are set to 'Event'.

In my case I am collecting Lightning strikes, and they have a position that is within a given area described by an Ellipse. I would really like to visualise this Ellipse "on the fly" with GeoEvent, but this bug is stopping me for now....

7 Replies
BrianLocke
Occasional Contributor II

Yeah Attempting to use this for generating ellipses for lightning data as well, when we run this in ArcMap and compare the output from the processor -- getting two very different outputs.  Thinking that there is a bug somewhere in the code for the GEO Ellipse Processor, also noticing that the linear unit is dependent on projected wkid for linear unit.  We are using geographic 4326, then to projected 3857, then back to 4326--everything is in kilometer.  Have started looking through the code on GitHub.  See if anything pokes out at me. Pink is Ellipse done in ArcMap using table to ellipse and the Green is GEP Ellipse Processor

ArcMap table to ellipseGEP Ellipse Processor settings

have tried 4326 with both of these--not even close. 

0 Kudos
ScottMoore__Olympia_
Esri Contributor

Hi Brian,

Is your input really coming in as 3857 coordinates?  I would think lightning strike data would come in with Lat/Long?  What happens if you make your input WKID 4326?

0 Kudos
ScottMoore__Olympia_
Esri Contributor

Also, when you measure your output axes, which is correct?  I am guessing the one from ArcGIS Pro/Desktop are, and the GeoEvent output is incorrect?  If so, does it seem like a unit conversion issue?  Here is the area of the geoevent code I would be looking at if you aren't already:

defense-solutions-proofs-of-concept/EllipseProcessor.java at a257842326f73f5691e56ba623c865146534c84... 

0 Kudos
BrianLocke
Occasional Contributor II

Good eye, I should have uploaded the proper screen captures, got to the point of trying everything, input wkid is 4326, as is the output wkid.  I do feel this is a units conversion issue.  I am looking at that exact method and thinking of attempting a couple of tweaks, then I will re-run -maven install then try again in gep.  

0 Kudos
RJSunderman
Esri Regular Contributor

Hey Brian -

First things first, let's make sure we're both looking at the same GitHub repository. The bug filed 02-Sept-2016 with Esri Support (BUG-000099124) was closed Dec-2017 with a remark 'Will Not Be Addressed'. Looking over its notes I see a link to an old Defense Solutions repo (no longer being maintained): solutions.arcgis.com/defense/templates/geoevent-extension

The repository currently being maintained is:  defense-solutions-proofs-of-concept/blob/master/geoevent-components

I asked Hanoch Kalmanovich‌ to compile a *.jar for me for the Ellipse Calculator (Defense Solutions is no longer maintaining pre-compiled JAR with the Git resources). He compiled the custom processor using the 10.5.0 GeoEvent SDK. I was able to add the JAR to a 10.6 release of GeoEvent Server without a problem.

When I attempted to reproduce the issue I found that ArcMap's GP Tool (Data Management > Table to Ellipse) produced consistent results with the custom processor I had added to GeoEvent Server. Here are my reproduction steps:

1) Create the following non-spatial table in a file geodatabase and run the 'Table to Ellipse' tool

  • Notice that the coordinate values are in a state plane projected coordinate system.
  • I chose a location in the northern part of Minnesota to emphasize elliptic distortion, if that was going to be a factor.

2) Publish the Ellipse feature class (above) as a feature service so I could add the feature to a web map for display.

3) Configure the following GeoEvent Service. I'll try to attach an XML of the GeoEvent Server configuration in case you want to take a look at input / output configurations or GeoEvent Definitions ... but GeoNet is making that difficult.

The purpose of this GeoEvent Service is to broadcast a received Point geometry to a stream layer, use the Ellipse Calculator to compute an area-of-uncertainty from received event attributes (semi minor axis, rotation angle, etc.) and then stream the Polygon geometry to a second stream layer. Notice that I have configured the processor to expect the input geometry in the state plane coordinate system appropriate for the area of interest and to use that same SPCS for the spatial analysis. (The output is GCS WGS 1984; the stream service would have projected the geometry to this SR in any case.)

4) Add the two stream services as new layer to the web map displaying the Ellipse feature service (created using ArcMap's GP Tool) ... then HTTP/POST the following JSON to the GeoEvent Server's input to see the geometries processed by GeoEvent overlaid on the web map.

[

  {
    "center_x": 697182.8,
    "center_y": 289173.3,
    "wkid": "3595",
    "semimajor": 1000,
    "semiminor": 375,
    "rotation_angle": 60,
    "id": "WKID_3595",
    "geometry": {
        "x": 697182.8,
        "y": 289173.3,
        "spatialReference": {
        "wkid": 3595
        }
    }
  }

]

Notice the coordinate values are expressed in the state plane projected coordinate system (WKID 3595). The only difference between this JSON data and the data in the non-spatial table is that the semimajor and semiminor values have been cut in half ... since the Ellipse Calculator wants the radius and the ArcMap's GP Tool takes the full diameter.

I allow the Ellipse Calculator to overwrite the received Point geometry with the polygon it computes.

Looking at the two ellipse polygon geometries in the web map, they align almost perfectly. Zooming in there is a discrepancy of about 12 meters where the two ellipse's don't align perfectly:

I'm going to chalk that difference up to projection of the geometries. The ellipse created by ArcMap's GP Tool was created with a native spatial reference appropriate for the area of interest (WKID 3595) whereas the ellipse created by GeoEvent Server's custom processor was calculated in that coordinate system, but then projected to GCS WGS 1984 for broadcast by the stream service. Both were then projected, on-the-fly, by the web map to Web Mercator for display.

Hanoch can dig into the Ellipse Calculator's implementation in GitHub if we still think there's a defect there. But it looks to me like the processor is working as its contributor intended. Please let me know if you disagree.

Cheers --

RJ

0 Kudos
RJSunderman
Esri Regular Contributor

It doesn't look like the thread's comments are going to allow me to attach a file. You can refer to the following 10.6 XML if you want to examine any property settings -- or copy/paste the code into a *.xml text file if you want to or try to import the GeoEvent Server configuration I illustrated above.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template productVersion="10.6.0">
    <geoEventServices>
        <geoEventService name="06c2a74b-bfaf-480a-8ba4-bda61c875b03" label="Compute AOU" description="" lineStyle="Diagonal" runningState="STARTED" statusDetails="">
            <inputs>
                <input ref="cbb16d04-039b-4954-8125-58177b71acb7" label="rest-json-in" left="51" top="77" width="75" height="40"/>
            </inputs>
            <outputs>
                <output ref="bfaa113f-4284-447d-8fd3-444962bd0379" label="EllipseBroadcast" left="470" top="250" width="103" height="40"/>
                <output ref="cad802bf-52cb-493f-8fb4-d2331340d550" label="file-json-out" left="471" top="173" width="79" height="40"/>
                <output ref="6cb3a7b7-2093-45d7-8de2-a545a8dca8f7" label="PointBroadcast" left="470" top="10" width="95" height="40"/>
            </outputs>
            <nodes>
                <node name="54e27398-c945-4b71-86dd-5054e9240847" label="(Ellipse) Stream Schema" left="210" top="210" width="200" height="40">
                    <processor uri="com.esri.ges.processor/FieldMapper/10.6.0">
                        <properties>
                            <property name="inputFields" type="String">id,geometry</property>
                            <property name="outputFields" type="String">id,geometry</property>
                            <property name="inputGeoEventDefinition" type="GeoEventDefinition">EllipseCalculator-Parameters</property>
                            <property name="outputGeoEventDefinition" type="GeoEventDefinition">StreamService-EllipseBroadcast</property>
                        </properties>
                    </processor>
                </node>
                <node name="fdd875ad-b7fc-4d63-8ee9-f954da753016" label="Calculate AOU" left="210" top="130" width="200" height="40">
                    <processor uri="com.esri.geoevent.solutions.processor.geometry/EllipseProcessor/10.5.0">
                        <properties>
                            <property name="rotationSource" type="String">Event</property>
                            <property name="rotationEvent" type="String">rotation_angle</property>
                            <property name="majorAxisRadius" type="Double">10000.0</property>
                            <property name="rotation" type="Double">0.0</property>
                            <property name="units" type="String">Meters</property>
                            <property name="wkidbuffer" type="Integer">3595</property>
                            <property name="minorAxisRadius" type="Double">5000.0</property>
                            <property name="wkidout" type="Integer">4326</property>
                            <property name="wkidin" type="Integer">3595</property>
                            <property name="minorAxisSource" type="String">Event</property>
                            <property name="majorAxisEvent" type="String">semimajor</property>
                            <property name="majorAxisSource" type="String">Event</property>
                            <property name="minorAxisEvent" type="String">semiminor</property>
                        </properties>
                    </processor>
                </node>
                <node name="f4d8ccab-55e2-47ea-83c7-9320e303ebba" label="(Center Point) Stream Schema" left="210" top="30" width="200" height="40">
                    <processor uri="com.esri.ges.processor/FieldMapper/10.6.0">
                        <properties>
                            <property name="inputFields" type="String">id,geometry</property>
                            <property name="outputFields" type="String">id,geometry</property>
                            <property name="inputGeoEventDefinition" type="GeoEventDefinition">EllipseCalculator-Parameters</property>
                            <property name="outputGeoEventDefinition" type="GeoEventDefinition">StreamService-PointBroadcast</property>
                        </properties>
                    </processor>
                </node>
            </nodes>
            <flow>
                <from ref="cbb16d04-039b-4954-8125-58177b71acb7" label="rest-json-in">
                    <to ref="f4d8ccab-55e2-47ea-83c7-9320e303ebba" label="(Center Point) Stream Schema"/>
                    <to ref="fdd875ad-b7fc-4d63-8ee9-f954da753016" label="Calculate AOU"/>
                </from>
                <from ref="54e27398-c945-4b71-86dd-5054e9240847" label="(Ellipse) Stream Schema">
                    <to ref="bfaa113f-4284-447d-8fd3-444962bd0379" label="EllipseBroadcast"/>
                </from>
                <from ref="fdd875ad-b7fc-4d63-8ee9-f954da753016" label="Calculate AOU">
                    <to ref="cad802bf-52cb-493f-8fb4-d2331340d550" label="file-json-out"/>
                    <to ref="54e27398-c945-4b71-86dd-5054e9240847" label="(Ellipse) Stream Schema"/>
                </from>
                <from ref="f4d8ccab-55e2-47ea-83c7-9320e303ebba" label="(Center Point) Stream Schema">
                    <to ref="6cb3a7b7-2093-45d7-8de2-a545a8dca8f7" label="PointBroadcast"/>
                </from>
            </flow>
            <geoEventsReceivedRateRange max="0.0" min="0.0"/>
            <geoEventsSentRateRange max="0.0" min="0.0"/>
        </geoEventService>
    </geoEventServices>
    <inputs>
        <input url="http://raistlin.esri.com:6180/geoevent/rest/receiver/rest-json-in" clusterCommand="START" connector="esri-receive-json-rest-in" connectorLabel="Receive JSON on a REST Endpoint" label="rest-json-in" name="cbb16d04-039b-4954-8125-58177b71acb7" runningState="STARTED" statusDetails="NULL" supportsAllGeoEventDefinitions="true">
            <adapter uri="com.esri.ges.adapter.inbound/Generic-JSON/10.6.0">
                <properties>
                    <property name="SpatialReferenceField" type="String"></property>
                    <property name="YGeometryField" type="String"></property>
                    <property name="CreateGeoEventDefinition" type="Boolean">false</property>
                    <property name="isLearningMode" type="Boolean">false</property>
                    <property name="ExistingGeoEventDefinitionName" type="GeoEventDefinition">EllipseCalculator-Parameters</property>
                    <property name="ZGeometryField" type="String"></property>
                    <property name="NewGeoEventDefinitionName" type="String">Generated-GeoEventDefinition</property>
                    <property name="asGeoJson" type="Boolean">false</property>
                    <property name="BuildGeometryFromFields" type="Boolean">false</property>
                    <property name="CustomDateFormat" type="String"></property>
                    <property name="JsonObjectName" type="String"></property>
                    <property name="XGeometryField" type="String"></property>
                </properties>
            </adapter>
            <geoEventDefinitionHistory>
                <guid>b70b6205-425f-4d45-a801-8baab650455e</guid>
            </geoEventDefinitionHistory>
            <geoEventsRateRange max="0.0" min="0.0"/>
            <supportedGeoEventDefinitions/>
            <transport uri="com.esri.ges.transport.inbound/HTTP/10.6.0">
                <properties>
                    <property name="headers" type="String"></property>
                    <property name="getRequestRawDataFieldName" type="String"></property>
                    <property name="acceptableMimeTypesServerMode" type="String">application/json</property>
                    <property name="clientURL" type="String"></property>
                    <property name="httpAppendToEnd" type="String"></property>
                    <property name="postContentType" type="String"></property>
                    <property name="clientPostBody" type="String"></property>
                    <property name="acceptableMimeTypesClientMode" type="String"></property>
                    <property name="httpMethod" type="String">GET</property>
                    <property name="httpTimeoutValue" type="Integer">30</property>
                    <property name="useLongPolling" type="Boolean">false</property>
                    <property name="clientURLProxy" type="String"></property>
                    <property name="frequency" type="Integer">5</property>
                    <property name="mode" type="String">SERVER</property>
                    <property name="honorLastModified" type="Boolean">true</property>
                    <property name="clientPostFrom" type="String">TEXT</property>
                    <property name="clientPostParameters" type="String"></property>
                    <property name="useClientURLProxy" type="Boolean">false</property>
                    <property name="clientParameters" type="String"></property>
                    <property name="getRequestIncludesRaw" type="Boolean">false</property>
                </properties>
            </transport>
        </input>
    </inputs>
    <outputs>
        <output clusterCommand="START" connector="esri-out-feature-stream-service" connectorLabel="Send Features to a Stream Service" label="EllipseBroadcast" name="bfaa113f-4284-447d-8fd3-444962bd0379" runningState="STARTED" statusDetails="The GeoEvent Definition &quot;FeatureRecord-Inbound&quot; is incompatible with the Stream Service JsonReceiver-EllipseBroadcast. There is at least one field in the GeoEvent that does not exist in the service." supportsAllGeoEventDefinitions="true">
            <adapter uri="com.esri.ges.adapter.outbound/JSON/10.6.0">
                <properties>
                    <property name="updateInterval" type="String">0.01</property>
                    <property name="flattenEvents" type="Boolean">true</property>
                    <property name="keepEventsUnique" type="Boolean">false</property>
                    <property name="mimeType" type="String">application/json</property>
                    <property name="prettyJson" type="Boolean">false</property>
                </properties>
            </adapter>
            <geoEventDefinitionHistory>
                <guid>03e85083-1c06-4dfa-b334-2619e1acd20d</guid>
                <guid>911e9c21-9053-4fee-9166-ff80449f0e61</guid>
                <guid>eb602ec9-6ebe-44ca-9327-d84107d5bfeb</guid>
                <guid>45eedabc-f6af-461c-9b52-348eebe27a54</guid>
                <guid>c212e56e-3589-43a8-b679-8422c9e97e85</guid>
            </geoEventDefinitionHistory>
            <geoEventsRateRange max="0.0" min="0.0"/>
            <supportedGeoEventDefinitions/>
            <transport uri="com.esri.ges.transport.outbound/StreamService/10.6.0">
                <properties>
                    <property name="path" type="ArcGISFolder">/</property>
                    <property name="datastore" type="ArcGISConnection">default</property>
                    <property name="geoEventDefinitionName" type="GeoEventDefinition"></property>
                    <property name="serviceName" type="ArcGISStreamService">JsonReceiver-EllipseBroadcast</property>
                </properties>
            </transport>
        </output>
        <output clusterCommand="START" connector="esri-out-json-file-log" connectorLabel="Write to a JSON File" label="file-json-out" name="cad802bf-52cb-493f-8fb4-d2331340d550" runningState="STARTED" statusDetails="NULL" supportsAllGeoEventDefinitions="true">
            <adapter uri="com.esri.ges.adapter.outbound/Generic-JSON/10.6.0">
                <properties>
                    <property name="mimeType" type="String">application/json</property>
                    <property name="prettyJson" type="Boolean">true</property>
                </properties>
            </adapter>
            <geoEventDefinitionHistory>
                <guid>b70b6205-425f-4d45-a801-8baab650455e</guid>
            </geoEventDefinitionHistory>
            <geoEventsRateRange max="0.0" min="0.0"/>
            <supportedGeoEventDefinitions/>
            <transport uri="com.esri.ges.transport.outbound/File/10.6.0">
                <properties>
                    <property name="outputFolderDataStore" type="FolderDataStore">8cb0049f-efa0-4fb4-8fd9-4372e6d848ad</property>
                    <property name="outputFilesShouldExpire" type="String">Delete</property>
                    <property name="outputFileExtension" type="String">json</property>
                    <property name="outputFileSizeLimitValue" type="Integer">1</property>
                    <property name="outputFileFooter" type="String">]</property>
                    <property name="outputBaseFileName" type="String">GenericJson-</property>
                    <property name="outputFileEventSeparator" type="String">,</property>
                    <property name="outputExpirationValueInMinutes" type="Integer">4320</property>
                    <property name="outputFileHeader" type="String">[</property>
                    <property name="outputFileFrequency" type="String">Monthly</property>
                    <property name="outputDirectory" type="String"></property>
                    <property name="outputFileSizeLimitMethod" type="String">By Time</property>
                    <property name="outputArchiveLocation" type="String"></property>
                    <property name="outputArchiveDataStore" type="FolderDataStore">8cb0049f-efa0-4fb4-8fd9-4372e6d848ad</property>
                </properties>
            </transport>
        </output>
        <output clusterCommand="START" connector="esri-out-feature-stream-service" connectorLabel="Send Features to a Stream Service" label="PointBroadcast" name="6cb3a7b7-2093-45d7-8de2-a545a8dca8f7" runningState="STARTED" statusDetails="The GeoEvent Definition &quot;FeatureRecord-Inbound&quot; is incompatible with the Stream Service JsonReceiver-PointBroadcast. There is at least one field in the GeoEvent that does not exist in the service." supportsAllGeoEventDefinitions="true">
            <adapter uri="com.esri.ges.adapter.outbound/JSON/10.6.0">
                <properties>
                    <property name="updateInterval" type="String">0.01</property>
                    <property name="flattenEvents" type="Boolean">true</property>
                    <property name="keepEventsUnique" type="Boolean">false</property>
                    <property name="mimeType" type="String">application/json</property>
                    <property name="prettyJson" type="Boolean">false</property>
                </properties>
            </adapter>
            <geoEventDefinitionHistory>
                <guid>911e9c21-9053-4fee-9166-ff80449f0e61</guid>
                <guid>03e85083-1c06-4dfa-b334-2619e1acd20d</guid>
                <guid>45eedabc-f6af-461c-9b52-348eebe27a54</guid>
                <guid>176035f0-9b44-4c50-9146-20a3bf0768e1</guid>
            </geoEventDefinitionHistory>
            <geoEventsRateRange max="0.0" min="0.0"/>
            <supportedGeoEventDefinitions/>
            <transport uri="com.esri.ges.transport.outbound/StreamService/10.6.0">
                <properties>
                    <property name="path" type="ArcGISFolder">/</property>
                    <property name="datastore" type="ArcGISConnection">default</property>
                    <property name="geoEventDefinitionName" type="GeoEventDefinition"></property>
                    <property name="serviceName" type="ArcGISStreamService">JsonReceiver-PointBroadcast</property>
                </properties>
            </transport>
        </output>
    </outputs>
    <geoEventDefinitions>
        <geoEventDefinition guid="b70b6205-425f-4d45-a801-8baab650455e" name="EllipseCalculator-Parameters" owner="admin" accessType="editable">
            <fieldDefinitions>
                <fieldDefinition name="center_x" type="Double" cardinality="One">
                    <fieldDefinitions/>
                </fieldDefinition>
                <fieldDefinition name="center_y" type="Double" cardinality="One">
                    <fieldDefinitions/>
                </fieldDefinition>
                <fieldDefinition name="wkid" type="String" cardinality="One">
                    <fieldDefinitions/>
                </fieldDefinition>
                <fieldDefinition name="semimajor" type="Double" cardinality="One">
                    <fieldDefinitions/>
                </fieldDefinition>
                <fieldDefinition name="semiminor" type="Double" cardinality="One">
                    <fieldDefinitions/>
                </fieldDefinition>
                <fieldDefinition name="rotation_angle" type="Double" cardinality="One">
                    <fieldDefinitions/>
                </fieldDefinition>
                <fieldDefinition name="id" type="String" cardinality="One">
                    <fieldDefinitionTag>
                        <name>TRACK_ID</name>
                    </fieldDefinitionTag>
                    <fieldDefinitions/>
                </fieldDefinition>
                <fieldDefinition name="geometry" type="Geometry" cardinality="One">
                    <fieldDefinitionTag>
                        <name>GEOMETRY</name>
                    </fieldDefinitionTag>
                    <fieldDefinitions/>
                </fieldDefinition>
            </fieldDefinitions>
        </geoEventDefinition>
        <geoEventDefinition guid="c212e56e-3589-43a8-b679-8422c9e97e85" name="StreamService-EllipseBroadcast" owner="admin" accessType="editable">
            <fieldDefinitions>
                <fieldDefinition name="id" type="String" cardinality="One">
                    <fieldDefinitionTag>
                        <name>TRACK_ID</name>
                    </fieldDefinitionTag>
                    <fieldDefinitions/>
                </fieldDefinition>
                <fieldDefinition name="geometry" type="Geometry" cardinality="One">
                    <fieldDefinitionTag>
                        <name>GEOMETRY</name>
                    </fieldDefinitionTag>
                    <fieldDefinitions/>
                </fieldDefinition>
            </fieldDefinitions>
        </geoEventDefinition>
        <geoEventDefinition guid="176035f0-9b44-4c50-9146-20a3bf0768e1" name="StreamService-PointBroadcast" owner="admin" accessType="editable">
            <fieldDefinitions>
                <fieldDefinition name="id" type="String" cardinality="One">
                    <fieldDefinitionTag>
                        <name>TRACK_ID</name>
                    </fieldDefinitionTag>
                    <fieldDefinitions/>
                </fieldDefinition>
                <fieldDefinition name="geometry" type="Geometry" cardinality="One">
                    <fieldDefinitionTag>
                        <name>GEOMETRY</name>
                    </fieldDefinitionTag>
                    <fieldDefinitions/>
                </fieldDefinition>
            </fieldDefinitions>
        </geoEventDefinition>
    </geoEventDefinitions>
    <tags/>
    <folderDataStores>
        <folderDataStore accessType="editable" canRead="true" canWrite="true" exists="true" label="GeoEvent_Output" name="8cb0049f-efa0-4fb4-8fd9-4372e6d848ad" path="C:\GeoEvent\output">
            <details></details>
        </folderDataStore>
    </folderDataStores>
</template>
BrianLocke
Occasional Contributor II

Thanks RJ for your detailed reply.  I had submitted two issues on github, one was for the original repo (depr) and the second was for the defense repo.  I have went ahead and closed one last week and forgot to get the second--it has now been closed.  Things are working as designed for the ellipse processor.  One issue that was my fault was simply using 4326 for input and 2992 for linear unit (Meters to Feet)  GeoEvents did not like this and was difficult to trouble shoot--I should have known better though.  Another is that ArcMap and Pro both were using the minor diameter and major diameter as the processor was expecting minor radius and major radius.  We're up and going and have been successfully generating proper ellipses.  We're successful in using wkid EPSG 6556 for in and linear unit and EPSG 2991 for in and Linear Unit we've been pushing out to 4326 but this is no longer necessary for many reasons and will be going to 2992--our agency standard.  Thanks for all of your help on this, same to Scott_Moore-esristaff

0 Kudos