Latest Contributions by RJSunderman

MBramer-esristaff‌ had to tackle this issue recently. I think what you need to do is use a Field Calculator to cast the Date to a Long integer value. Use an expression like myDateAttributeField + 0 and write the value into a new field whose type is Long (not Date). Then you can use a second Field Calculator to compute a Long integer representing epoch milliseconds five minutes ago:  currentTime() - (5 * 60 * 1000). Write this value into another new field whose type is Long. Then use a filter to compare the two integer values. - RJ ... View more

John - I wanted to update you to let you know that enhancements are being rolled into the 10.5 release for the Intersector processor. At 10.4.x the assumption was that the output geometry would be the same dimension as the event's geometry. That's why the illustration in the product introduction tutorial worked and showed the intersection of an event's polyline with a polygon GeoFence output as a polyline. Beginning with the 10.5 release, you will be able to configure an Intersector processor to output a specified 'Output Geometry Type'. So, in your case, if you are processing an event with an associated polyline geometry and you want to obtain the point at which your event's polyline intersects a polyline GeoFence - you will be able to do that. Illustrated below are a couple of the use cases I've used to verify the enhancements to the Intersector processor at 10.5 You'll notice that you can still obtain a multipart polyline when an event's polyline intersects a polygon GeoFence at two separate line segments. You can also get the point intersection between two polylines. There are several other test cases which I will try to lay out in a blog, which will serve as the documentation for the enhancements to the Intersector processor. Hope you find the 10.5 enhancements useful - RJ ... View more

When conducting our benchmarks for the upcoming 10.5 release we encountered an unexpected drop in the raw number of events per second we can ingest in a TCP/Text input and send out a TCP/Text output (without conducting any processing or filtering on the event data). The issue turned out to be that we had accidentally staged the software on a different type of Amazon EC2 than what we had used in previous benchmark tests. It turns out that the c4.2xlarge class supports Intel® AVX2  whereas the c3.2xlarge class does not. Support for AVX2 can double the number of floating-point operations per second an application can perform. For our benchmarks that meant that event throughput for simple events which included coordinates for a point geometry fell from over 4,000 events per second to well below 3,000 events per second. Instance Type vCPU Memory (GiB) Storage (GB) Networking Performance Physical Processor Clock Speed (GHz) Intel AVX† Intel AVX2† Intel Turbo EBS OPT Enhanced Networking† c3.2xlarge 8 15 2 x 80 SSD High Intel Xeon E5-2680 v2 2.8 Yes - Yes Yes Yes c4.2xlarge 8 15 EBS Only High Intel Xeon E5-2666 v3 2.9 Yes Yes Yes Yes Yes I mention this because, looking at the two class specifications in the table above, it would be very easy to overlook something "minor" like support for Intel AVX2 technology. (Scroll the table display to the right to see that the "C3" instance does not support AVX2.) As with our discovery that different types of RAM (e.g. DDR3 vs. DDR4) are very significant - capabilities supported by your chosen server's processor can be very significant. Cross Reference: - https://aws.amazon.com/ec2/instance-types - http://www.intel.com/content/www/us/en/benchmarks/performance-xeon-e5-v3-advanced-vector-extensions-paper.html ... View more

Christine Wiltawsky​​ - Yes, I just verified using a system configured with 10.4 Patch1. Note that I've chosen to blur irrelevant property values since, in my example, I'm writing to an existing field of type Date. ... View more

Robert - I cannot comment on why publishing your survey to Portal vs. ArcGIS Online would produce the differences in schema you are observing. If you want to try and dig into why GeoEvent is receiving event records from Portal which have not been recently updated (when polling an ArcGIS Online feature service appears to work), you will need to configure GeoEvent to include DEBUG messages in its log. You might try setting DEBUG logging on just the following loggers within GeoEvent Manager.  If you see errors or warnings associated with these, please let us know. com.esri.ges.datastore.agsconnection.DefaultAGOLConnection com.esri.ges.datastore.agsconnection.DefaultArcGISServerConnection com.esri.ges.datastore.agsconnection.DefaultPortalConnection Unfortunately the messages you need which include the WHERE clause GeoEvent uses to identify event records which have been recently added or changed is provided by HTTP client log messages. I say this is unfortunate because that particular component is very, very chatty. You can only imagine as GeoEvent is communicating via HTTP / REST with all sorts of stuff ... If you set DEBUG logging on the GeoEvent <root> component -- which you can do for a short period of time, just long enough to catch your input polling and receiving newly updated records -- you can catch the HTTP client log messages and review exactly what GeoEvent is requesting from the feature service it is polling. You will want open the karaf.log in a text editor and search for a logged message with the following substrings: Executing following request: GET http://services.arcgis.com arcgis/rest/services/ your_feature_service_name /FeatureServer/ layer_index /query where=1%3D1+and+LastUpdated+%3E%3D+%272016-05-10+21%3A34%3A07 (decoded: where=1=1 and LastUpdated >= '2016-05-10 21:34:07) That last message is URL encoded and can be difficult to read, but that's how it's going to appear in the log file. You can copy/paste the URL encoded string into an online decoder (I like using the URL Decoder/Encoder​) to get the decoded string I included above. Notice that the decoded string contains a date/time allowing GeoEvent to specify that any feature returned should have a date/time greater than that value. Crawling through the karaf.log is going to be a chore, but since you know the name of the feature service being polled and the name of the attribute field suppling the date/time value you should be able to zero in on messages of interest fairly quickly (click the illustrations below to open the thumbnails): If you filter your karaf.log down to just messages with "Executing following request" and "Got response from HTTP request" you should start to see a pattern, at least in the ArcGIS Online example. This pair of messages is requesting information on the feature service: This pair is telling GeoEvent that there are no features which satisfy the query: Eventually, when there *are* features which satisfy the query, you should see messages like the following: This third example is interesting because GeoEvent is following its query for features WHERE LastUpdated+%3E%3D+%271970-01-01+00%3A00%3A00.000%27 (e.g. LastUpdated >= '1970-01-01 00:00:00.000') with a query requesting only the OBJECTIDs be returned (e.g.  returnIdsOnly=true). GeoEvent has to do this so that it knows whether it got all of the features or whether a feature service is limiting the number of features returned in response to a client's query. You might be asking "Why the date/time back in 1970??"   When an input which polls for incremental updates is first created a false date/time is incorporated in the query’s WHERE clause. This is because the new input has not had a chance to cache a date/time value and needs to make one up which will likely get everything the feature service has. Thereafter GeoEvent will substitute the greatest date/time value from the set of features returned in response to its query ... and the next query's WHERE clause will ask for only the features whose date/time is strictly greater-than the previous observed date/time. Hopefully all that technical background and log file illustrations were helpful. You, and possibly whomever you work with in Esri Tech Support, will need that background on what GeoEvent is doing in order to diagnose why the query to the Portal item is not returning just the recently updated event records. - RJ ... View more

Hello Rao – Yes, the XML sample you provided above can easily be adapted by the product’s XML adapter and processed by a GeoEvent Service. The first remark I have is that XML is translated to JSON by the product’s adapter.  So you will want to avoid using XML attributes which can contend with sub-elements of a given node. Your data sample is pretty clean, but to illustrate, this is what it looks like as JSON: Your challenge, using GeoEvent, will be that you want to access items in a list which is part of a larger structure. If you were to stipulate that the list <Assetstatus> would always have exactly 5 items, then we could assume that you could pull specific items out by index. For example: ez_data.fz.Statuslist.Assetstatus[3].Status This isn’t normally something that you can stipulate and we cannot therefore assume there will be a fixed number of items in the list. In that case you have one opportunity – as part of the GeoEvent input’s configuration – to tell the adapter that it should use something other than <ez_data> as the root node. You specify this using the XML Object Name parameter: This way, the five items in your sample XML’s <Assetstatus> list are each brought in as separate events. The upside is that you don’t need to worry about how many items will be in the list. The downside is you do not have the title, id, timestamp, or coordinate information that is above the <Assetstatus> node in the data structure. There is work you can do to pull information from a XML feed’s header and incorporate it into individual event records. Please refer to the thread XML can I use this source​ if you are interested in taking that approach. Hope this information is helpful - RJ ... View more

Jon – adding to Sagar's advice above, I noticed that you said that your "SSL Cert is set up on the machine, registered with arcgis server and binding configured with IIS". Did you import the SSL certificate both into Server (using the Server Admin Directory) and also into Portal (using the Portal Admin Directory)? I think your IIS certificate needs to be imported into both products. – RJ ... View more

Sardar - Please take a look at the following threads: Geoevent Processor Geofence Question​ Overlapping geofences with different attributes​ What you are asking for GeoEvent will not do.  The only way to load a single GeoFence representing an aggregate of several Geometries into GeoEvent would be to create a convex hull (or envelope) around the points and import that polygon Geometry as a GeoFence. You cannot maintain the identities of several different Geometries as part of a single GeoFence. Every GeoFence must have a single Geometry which can be assigned to a category. Within a category every GeoFence must have a unique identifier / name. I cannot think of a way, using GeoEvent, to determine the “closest” GeoFence to a Geometry associated with a received event. The event’s Geometry either does or does not have a spatial relationship with a GeoFence (e.g. intersects the GeoFence’s Geomtery). - RJ ... View more

Hello Sardar – The Poll an External Website for JSON input connector does indeed have a property ‘Receive New Data Only’ … but the property does not provide the behavior I think many folks are looking for. When you make a request on a site’s endpoint, unless the site is configured to accept a parameter as part of your request specifying the data you want, the only way GeoEvent has to determine if the site has “new” data is to look for a specific header value in the headers returned in the response from the external server. This “Last-Modified” header (described in Section 14 of RFC 2616 “Header Field Definitions” of the HTTP/1.1 protocol) tells GeoEvent to go ahead and parse the response from the server as there is new information in the response. From the spec: 14.29 Last-Modified The Last-Modified entity-header field indicates the date and time at which the origin server believes the variant was last modified. Last-Modified = "Last-Modified" ":" HTTP-date An example of its use is Last-Modified: Tue, 15 Nov 1994 12:45:26 GMT The exact meaning of this header field depends on the implementation of the origin server and the nature of the original resource. For files, it may be just the file system last-modified time. For entities with dynamically included parts, it may be the most recent of the set of last-modify times for its component parts. For database gateways, it may be the last-update time stamp of the record. For virtual objects, it may be the last time the internal state changed. What "Last-Modified" does not do is key in on specific records which have changed. For example, a site serving information on a dozen different active wildfires might update a few of its wildfire event records and then indicate that update(s) have been made by setting its “Last-Modified” to date/time the updates were committed. When GeoEvent polls it will see that the "Last-Modified" date/time is later than the value it has cached and will ingest the event data returned from the site (all of the wildfire records in this case). The Poll an ArcGIS Server for Features input, on the other hand, is a little more flexible. That input also has a property for polling only for feature records recently added or updated – but in this case GeoEvent is able to work with the Esri Feature Service to incorporate a WHERE clause in its request in order to receive only the specific records which satisfy the WHERE clause. This is not functionality provided by the input you’re using which is making a more general request on a less well-known external server. It is difficult in the 10.3.x / 10.4.x releases to create a filter expression to compare an event’s date/time against another date/time value. You might be able to implement a work around by handling date/time value received from the external server as Long Integer values (in epoch milliseconds) rather than as Date values. I’ve included a couple of references below to discussion threads which mention this. Generally, however, using a GeoEvent Service to detect and discard “old” event data is not recommended. GeoEvent is assuming that all of the data either sent to an input via HTTP/POST or polled from an external server is “live” and should be processed as current real-time values. See Also: How to Create Temporal Filter in GeoEvent​​ Filter out Time Anomalies​ ... View more

Hello Stephan – Sorry your question has gone so long without a response. Is the issue you describe still a problem for you? You might look at the thread Re: Spatiotemporal big data store isn't showing up in Geoevent Manager which Martin and I contributed to March 5th (a few weeks after you posted this question - sorry). It sounds to me like maybe you don’t have a connection to Portal registered as a GeoEvent Data Store? The GeoEvent Data Store connection registered as ‘Default’ is not sufficient to begin using the new spatiotemporal big data store – you have to register a “Portal” type connection as a GeoEvent Data Store and provide username and password credentials enabling GeoEvent to authenticate with your Portal. Once you do this, and then navigate to Site > GeoEvent > Spatiotemporal Big Data Stores in GeoEvent Manager, you should see an entry for an enabled BDS in the Spatiotemporal Big Data Store pulldown toward the top of the Manage Spatiotemporal Big Data Stores page. - RJ ... View more

Sharon - Sorry your questions went for so long without a reply. Are the above still issues for you?  Did you perhaps get some help from Esri Tech Support? You shouldn’t have any problems deleting a GeoEvent Definition created by a processor you’ve incorporated into a GeoEvent Service. Every time I’ve done this the processor has recreated whatever event definition it thinks it needs. Generally I advise folks to copy, rather than edit, a GeoEvent Definition they didn’t create. In this case, because an event definition “upstream” from the processor changed, you could signal the processor to clean-up its managed event definitions by making a simple change to the GeoEvent Service – like removing a connecting line from between two nodes – then publishing the changes to the GeoEvent Service. You can then restore the connecting line you deleted and publish the GeoEvent service again … processors like Field Calculators, GeoTaggers, and Field Enrichers which created event definitions should delete them as part of the first publication and create them once events are received (after the second publication). If the input’s event count is increasing then I’d assume that event data is being received and successfully adapted to create a GeoEvent. If the GeoEvent Service’s ‘In’ event count is increasing, but not the ‘Out’, I’d assume that events are either being filtered or are dying when a processor node attempts some action and an exception is thrown. On the other hand if neither the ‘In’ event count nor the ‘Out’ event count are increasing I would assume that events are being placed on a queue by the running inbound connector (input) but never being taken off because the RabbitMQ message broker is not running. - RJ ... View more

Simon - I'm going to use the 10.3.x-r3 release of the introduction tutorial (the version marked July 2015 and Rev B) with our current 10.4.1 release candidate to try and answer your question about what needs to change in the module 2 narrative and exerices to bring them into line with 10.4.x ... On page 28, step 8: Click to edit the field tagged GEOMETRY and rename the field Location. You should leave the field named geometry as that is what the 10.4.x release is going to want to see when using the Latitude and Longitude fields to build a point Geomery for you from the Double values received by the TCP/Text input. The field named geometry must still be moved to make it the last field in the GeoEvent Definition. On page 35, step 3: Use the illustration below to configure the properties of the new processor. You should configure your Field Mapper processor to map the field name you retained (geometry from Flights-TcpTextIn) to the field named geometry (from the Flights event defintion imported from your published feature service's layer). Your output from the Flights GeoEvent Service should mirror what is illustrated on page 36: Flights-TcpTextIn,SWA2706,18-Apr-2016 15:21:59,IAD,TPA,B733,37000,-79.585739,34.265521,"-79.585739,34.265521" Flights,,SWA724,18-Apr-2016 15:29:20,IAD,ABE,SF34,10000,"-76.405289,39.573271" Flights,,SWA992,18-Apr-2016 15:29:34,IAD,MDW,B737,36400,"-82.067414,39.630957" The two most recently simulated events are no longer associated with the Flights-TcpTextIn event defintion, they have been mapped to the Flights event definition you imported from your published feature service's layer. The double set of commas correspond to the OBJECTID field left unmapped in your Field Mapper Processor. The Geometry (the last field in the Flights event defintion) holds the point generated for you by the TCP/Text input; its value is represented as a quoted pair of Double values. Those are the only changes I need to make to the July 2015 and Rev B version of the introduction tutorial to accomodate the 10.4.x behavior I was blogging above. I have a goal of walking through the entire product introduction tutorial before the UC this summer and updating its narrative and illustrations to reflect the upcoming 10.4.1 product release. - RJ ... View more

Daniel - For the 10.3.1 release we tried incorporating a library to improve the user's sign-in experience but discovered a rather severe security vulnerability and had to revert (remove) the code changes. We improved single-sign-on experience for the 10.4 product release. To provide you a little bit of background, GeoEvent requires anonymous access to two different REST endpoints in order to coordinate authentication with Portal when Portal has been federated with ArcGIS Server and authentication is passing through the web-tier (e.g. Basic Authentication or IWA). https://localhost:6443/arcgis/rest/info (alternatively http://localhost:6080/arcgis/rest/info) Provides ArcGIS Server Authentication Information https://portal-server-hostname/arcgis/sharing/generateToken Portal’s token generation page At 10.3 / 10.3.1 ... if GeoEvent cannot access these endpoints its failover behavior is to prompt the user to acquire and provide a token in order to log-in to the GeoEvent Manager. I think the 10.4 / 10.4.1 login experience, when federated with Portal, will be more in-line with what you expect.  There is one known issue that is currently deferred to 10.5 -- after federating with Portal and completing the configuration of your SSL certificates you often have to restart the GeoEvent Windows service twice (2x) in order to get it to recognize the new security topology. Hope this information is helpful - RJ ... View more