Latest Contributions by RJSunderman

Eric Ironside‌ posted a follow-on to Andy Ommen‌'s blog (above) ... and included updates to Andy's code. Check out:  https://community.esri.com/people/eironside-esristaff/blog/2019/03/21/scripting-tasks-using-the-geoevent-admin-api-update-inputs?sr=search&searchId=200fe429-cdea-4861-8e01-92c5a10dcd1c&searchIndex=2‌ ... View more

Hello Nick - The reference you highlight, that stream services only run on a single machine, is intended to convey that the JVM used to run GeoEvent Server is the container used to run stream services (unlike traditional map/feature services which have multiple instances and are run by SOC components managed by ArcGIS Server). So the server machine used to publish a stream service is the machine whose GeoEvent Server JVM is running the stream service. However, as I mention above, copies of the Esri Feature JSON records a stream service broadcasts are forwarded to other GeoEvent Server instances in an ArcGIS Server site. This “fan out” allows web clients to subscribe to any machine’s stream service and get all of the feature records processed across a single site, multiple machine solution – regardless of which machine received an event record and which machine actually processed the event record. A bug (BUG-000114373) with the "fan out" mechanism was addressed with the 10.6.1 release and ported back for release as part of ArcGIS GeoEvent Server 10.6 Patch 2. The 10.7 release does not include any specific changes that, I think, should influence an system architect to choose a "silo" approach (where multiple independent instances of GeoEvent Server are run, each in their own ArcGIS Server site) vs. a "site" approach whose architecture deploys multiple GeoEvent Server instances configured as part of a single ArcGIS Server site. If scalability is your primary objective, I would probably recommend the "silo" approach. The solution you architect would need to include an external Apache Kafka (or similar message broker) to handle event record distribution across the multiple independent ArcGIS Server / GeoEvent Server instances. Adopting and maintaining your own Kafka solution introduces its own technical burden, but I think we are finding that, for scalability, this approach lends itself to a system that is easier to maintain and administer overall. If reliability and fault-tolerance is your primary objective (I really dislike using the term high-availability) then a "site" approach is an option. The solution you architect could deploy multiple ArcGIS Server / GeoEvent Server instances in a single site and rely on the Apache Kafka and Zookeeper built-in to the GeoEvent Gateway to mitigate individual machine failures. But a reliability objective can also be addressed with an an active / active approach using multiple independent instances of GeoEvent Server to build redundancy into a distributed solution for fault-tolerance. There are advantages to going with the single site, multiple machines approach for resiliency. Specifically when your solution relies on GeoEvent Server polling external web servers / web services for data (which you mentioned), we have observed that the GeoEvent Gateway reliably mitigates the failure of a single node -- which had been the node polling for input -- by allowing another node to "adopt" the input and begin handling the data polling activity. There are also significant system complexity and administration disadvantages to the "site" approach, which is why I recommend folks who are considering multiple machine, distributed architectures for real-time solutions work with their Esri Technical Advisor or contract with Esri Professional Services for consultation. Only after discussing specific objectives and weighing both the pros and cons can a recommendation be made as to which approach your solution ought to take. - RJ ... View more

Hello Cami – There are a few things I might suggest you try as you work to troubleshoot your issue: Reduce your geofence synchronization rule's polling frequency. Try setting the synchronization to run once every 60 seconds rather than once every second. It’s understandable that you want the polygon buffers created around designated points to be imported as geofences as quickly as possible so that polling on your ex2 feature dataset can take immediate advantage of the new geofences. A frequency of 1 second is too aggressive for geofence synchronization however. The synchronization rule has a simple timer and, every time the timer expires, a REST request is made against the feature service to retrieve its polygon feature records and import them to update the GeoEvent Server catalog of registered geofences. These requests can be relatively expensive and depending on your network, your database server, and other factors it may realistically take longer than 1000 milliseconds to retrieve the records, update the geofence catalog, and advise the various spatial processors in any GeoEvent Services you have configured that the geofence cache has been updated. (Individual processors cache some of the information about newly registered geofences so that they don’t have to each interact with the geofence catalog every time an event record is received by the processor for processing. This "chatter" between the geofence catalog and your GeoEvent Services tends not to take a lot of time, but it does take some non-zero amount of time.) When asserting the geofence synchronization changes, watch your system clock and try to click the ‘Synchronize’ button more-or-less as your system clock advances to the next whole minute. I’ve found that, when debugging, it helps to know when the geofence synchronization rule is firing. Since this is an asynchronous timer and the GeoEvent Manager doesn’t provide you with any visual indication that the synchronization’s timer is about to expire and a new request is about to be made to update geofences, if I set the synchronization interval to 60 seconds and start the timer (by clicking ‘Synchronize’) just as my system clock is advancing from 57, 58, 59, to 60 … I have a pretty good idea of when synchronizations will occur. You can then refresh your web page displaying known geofences and check to see if in fact new geofences are created/updated when your GeoEvent Service processes feature records polled from the ex1 feature dataset, discovers an attribute value set to “true”, and creates a buffer around the event record's geometry. As part of your debugging, stop the input which is polling the ex1 feature dataset, wait for a synchronization to occur, then start the input and allow it to conduct exactly one poll. Something to keep in mind is that this entire workflow has inherent race conditions with asynchronous timers that you don’t have a lot of visibility into or control over. If feature records are polled from the ex1 feature dataset every 20 seconds, there is a chance that the GeoEvent Service is busy creating the buffer and making a request to update polygon features in the buffer feature dataset at the same time the synchronization rule polls to request feature records from that same dataset (to update geofences in the catalog). If your debugging allows you to know that a synchronization will fire at the top of every minute and you wait perhaps 10 seconds to make sure the synchronization rule has a chance to complete before you start the ex1 input to poll features one time and stop the input after you confirm that all 25 (or whatever) feature records were polled, you can quickly query the buffer feature service yourself via the ArcGIS REST Services Directory to confirm that expected polygon feature records were actually created/updated and wait the balance of the minute for the next geofence synchronization rule to fire so that you can check to see if the polygon features you know to exist were successfully imported as geofences. Notice that nowhere in this debugging step did I start the input which is polling the ex2 feature dataset. That’s yet another asynchronous timer that you must deal with when debugging. On one hand, I want to make sure that all the pre-requisite steps have occurred successfully – that all ex1 features are being polled, expected buffer polygons are created/updated, and expected geofences established – before I start ex2 polling for additional features to determine if there a spatial coincidence exists between some other set of feature records and synchronized geofences. On the other hand ... Make sure that the input responsible for polling the ex2 feature dataset actually ingests all of the feature records exposed by that feature service. This was the heart of your question. It’s easy to focus on expected field calculations which are not occurring as you expect … because a filter configured to identify GEOMETRY INSIDE ANY buffer/.* is not providing any event records to a Field Calculator … even when you are reasonable sure geofences the filter should be using exist. Regardless of whether the pre-requisite steps above have occurred you should be able to start the input responsible for polling the ex2 feature dataset and confirm that every 10 seconds (or whatever) the input actually polls all feature records exposed by that feature service. This should happen regardless of whether the ex1 input is running or stopped or whether the synchronization rule is updating geofences in the geofence catalog the way you expect. You are correct that if the input is configured with ‘Get Incremental Updates’ set to ‘No’ and the ‘Query Definition’ beneath advanced has its default ‘1=1’ then all feature records in the feature dataset should be polled by the input every polling interval for processing. You can use the event counter on the GeoEvent Manager to confirm the number of event records adapted by the ex2 input and even create an output to log the event records created by the input as CSV Text or JSON to a system file immediately after the input before they are sent to a spatial filter or other processor. You want to determine that the input is indeed polling feature records from ex2 as you expect and that you can examine the data for these event records in a system *.json or *.txt file before you worry too much about what processing you want to perform on these feature records. Make sure your geofences do not have an associated ‘start time’ or ‘end time’. This is an easy thing to overlook. As you debug to confirm that geofences are being created from the buffer feature records, hover over the geofences in GeoEvent Manager’s list. Make sure the start time and end time field are empty/blank. Even if an event record’s geometry intersects a geofence spatially if the date/time of the event record does not intersect the geofence temporally your ex2 event records will not pass through your filter. Think of it like this – an event record must be proven to intersect (or fall inside) a geofence for the expression GEOMETRY INSIDE ANY buffer/.* to return TRUE. If the filter cannot find any geofence whose start/end time range intersects the date/time of the event record being tested, the event record’s location / geometry does not matter. There are no temporally relevant geofences, so it cannot be proven that the event record’s geometry is inside any particular geofence, and therefore no event records pass through the filter. I encourage you to open a support incident with Esri Technical Support so that an analyst can work with you to examine your configured input(s), GeoEvent Services, and geofence synchronization. There's potentially more to the workflow you've described that is apparent on the surface. I hope the information above is helpful in finding the problem. Best Regards – RJ ... View more

Hey Nate, Yes, if you would please submit a technical support incident, that will help get a bug report formally documented. If you have JSON data we can send to a receive JSON input that shows reproducability, that would also be a huge help. - RJ ... View more

Hello Nathan - I've not seen the error you are reporting before. If the inbound connector (input) is able to receive and adapt data to create an event record, and these event records process through a GeoEvent Service, and data can be logged as text to a CSV file ... then my understanding is that event records have undergone several Avro serialization / deserialization cycles. We cannot tell, from the screen capture you provided, whether the INFO message being logged by the ges.messaging.jms encoder is in response to a problem encountered by an inbound connector's transport or adapter, if the message is being logged because a processor you've configured is unable to handle event records it has received, or if -- as you suggest -- the error is coming from a failure to add or update feature records through a feature service. My first step would be to remove any processors or filters from my GeoEvent Service and see if I can successfully ingest event records and log their data as JSON in a system file. I prefer the Write to a JSON File output as the JSON format supports hierarchy and multicardinality that delimited text (e.g. CSV) does not. Also, you’re probably aware, that event records must be “simplified” to a flat structure without any hierarchy or multicardinality before the event records can be sent to an output tasked with adding or updating feature records through a feature service. This step, I think, will help us figure out whether all of the data being sent to GeoEvent Server is being processed through to an output, or whether some portion of the data is being rejected on the inbound side. Please open an incident with Esri Technical Support so that an analyst can be assigned to work with you and track the steps being taken to address the issue. If this ends up being a bug in an input, processor, or output where a certain type of data is not being handled property, the product team will need information from technical support before we start work to identify a root cause. Best Regards – RJ ... View more

Anna, GeoEvent Server is fundamentally RESTful, by which I mean the most reliable integration tends to leverage inputs where GeoEvent Server hosts a REST endpoint to which data providers can HTTP/POST data as JSON, geoJSON, XML, or delimited text ... or inputs which periodically poll an external web server / service which then sends event records back formatted as JSON, geoJSON, or XML. Both of these types of inputs are available out-of-the-box without any custom development (e.g. programming). When Robert suggests that you'll still need to develop something to relay data to the GeoEvent Server, I suspect that he is thinking of a RESTful API. If a company whose business is deploying sensors for system monitoring wants to make it easy to integrate with ArcGIS Enterprise and GeoEvent Server, they'll develop an REST API which either allows an external web client (e.g. GeoEvent Server) to poll an endpoint or allows some way to subscribe and begin receiving periodic data pushes from the API to a REST endpoint. - RJ ... View more

Hello Brian - I do not have an ArcGIS Enterprise I can easily configure with SQL Express ... so I'll have to show you the debugging steps necessary to generate log messages which will (hopefully) tell us what the issue is. It may be that the query parameters generated by GeoEvent Server's output and passed with the HTTP request to the feature service are not compatible with the SQL Express database. But before we go there, let's check to make sure the date/time values that are being written into the feature record in the database are value we expect. To set-up this test I created a feature service whose schema matches the following GeoEvent Definition. Notice that my attribute field reported_dt is a Date. If the field is not a Date the Field Calculator will not write the correct value into the field when evaluating its expression receivedTime() and the GeoEvent Server's output will not be able to use the field when requesting features be deleted. Here's the GeoEvent Service I configured to ingest data with a String/TrackID and build a Geometry from two Double values Latitude and Longitude: Notice the Field Mapper is preparing my event structure to match the structure and attributes of feature records expected by the feature class. The field reported_dt is left unmapped because the Field Calculator is writing the value receivedTime() into that field. I could also have used the Field Mapper to map the event property $RECEIVED_TIME into that field and elected to not use the Field Calculator at all - but you indicated in your post that you wanted to use the Field Calculator's function to obtain the event record's received date/time. At 5:32 (and 20 seconds) by my server's clock, I sent an event record to GeoEvent Server and observed the following feature record created in my feature class: "features": [   {    "attributes": {     "oid": 401,     "track_id": "VWY-86-ABU",     "reported_dt": 1538094740000    }   } ]‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ Note that the date/time recorded in the feature class is not my server's local time, it is an epoch long integer whose value is UTC / GMT. This is important because whatever value is passed as part of a deleteFeatures request will have to reflect the UTC time, not my server's local time. (The value 1538094740000 is 12:32:20 am the next day, seven hours ahead of my current local time, 5:32:20 pm...) Your configuration of your output looks correct. For my test I chose to have the output delete any feature records which are older than two minutes, attempting to delete "old" records every 30 seconds: If I turn DEBUG logging on for GeoEvent Server, every 30 seconds I should see a block of log messages similar to the following. Unfortunately the information we want to capture is logged as part of the HTTP Client ... and there is a lot of HTTP traffic logged when you turn on DEBUG logging for that component, so I'll try to bullet for  you the information from the log messages below that you need to look for:   2018-09-26T17:32:53,495 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | Http                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Adding parameter (f/json). 2018-09-26T17:32:53,495 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | Http                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Adding parameter (token/xxxxxxxx). 2018-09-26T17:32:53,495 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | Http                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Adding parameter (rollbackOnFailure/true). 2018-09-26T17:32:53,495 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | Http                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Adding parameter (where/reported_dt < timestamp '2018-09-27 00:30:53'). 2018-09-26T17:32:53,498 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | Http                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Executing following request: POST https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures HTTP/1.1 2018-09-26T17:32:53,499 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | RequestAddCookies                | 52 - com.esri.ges.framework.httpclient - 10.6.0 | CookieSpec selected: default 2018-09-26T17:32:53,499 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | RequestAuthCache                 | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Auth cache not set in the context 2018-09-26T17:32:53,499 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | PoolingHttpClientConnectionManager | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connection request: [route: {s}->https://your-machine.domain:443][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20] 2018-09-26T17:32:53,500 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | PoolingHttpClientConnectionManager | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connection leased: [id: 43][route: {s}->https://your-machine.domain:443][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20] 2018-09-26T17:32:53,500 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | MainClientExec                   | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Opening connection {s}->https://your-machine.domain:443 2018-09-26T17:32:53,500 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | DefaultHttpClientConnectionOperator | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connecting to YOUR_MACHINE.DOMAIN/10.27.102.67:443 2018-09-26T17:32:53,501 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connecting socket to YOUR_MACHINE.DOMAIN/10.27.102.67:443 with timeout 0 2018-09-26T17:32:53,502 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2] 2018-09-26T17:32:53,502 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, ... 2018-09-26T17:32:53,502 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Starting handshake 2018-09-26T17:32:53,513 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Secure session established 2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 |  negotiated protocol: TLSv1.2 2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 |  negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 |  peer principal: CN=your-machine.domain, OU=Business Development, O=Esri, L=Redlands, ST=California, C=US 2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 |  peer alternative names: [your-machine.domain] 2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 |  issuer principal: CN=ESRI Enterprise Root, DC=empty, DC=local 2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | DefaultHttpClientConnectionOperator | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connection established 10.27.102.67:60152<->10.27.102.67:443 2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | MainClientExec                   | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Executing request POST /server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures HTTP/1.1 2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | MainClientExec                   | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Target auth state: UNCHALLENGED 2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | MainClientExec                   | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Proxy auth state: UNCHALLENGED 2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> POST /server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures HTTP/1.1 2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> Content-Type: application/x-www-form-urlencoded 2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> charset: utf-8 2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> Referer: https://your-machine.domain:6143/geoevent/admin/datastores/agsconnection/default 2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> User-Agent: GeoEvent Server 10.6.0 2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> Content-Length: 314 2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> Host: your-machine.domain 2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> Connection: Keep-Alive 2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> Accept-Encoding: gzip,deflate 2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "POST /server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures HTTP/1.1[\r][\n]" 2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "Content-Type: application/x-www-form-urlencoded[\r][\n]" 2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "charset: utf-8[\r][\n]" 2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "Referer: https://your-machine.domain:6143/geoevent/admin/datastores/agsconnection/default[\r][\n]" 2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "User-Agent: GeoEvent Server 10.6.0[\r][\n]" 2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "Content-Length: 314[\r][\n]" 2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "Host: your-machine.domain[\r][\n]" 2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "Connection: Keep-Alive[\r][\n]" 2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "Accept-Encoding: gzip,deflate[\r][\n]" 2018-09-26T17:32:53,517 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "[\r][\n]" 2018-09-26T17:32:53,517 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "f=json&token=xxxxxxxx&rollbackOnFailure=true&where=reported_dt+%3C+timestamp+%272018-09-27+00%3A30%3A53%27" 2018-09-26T17:32:53,630 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "HTTP/1.1 200 OK[\r][\n]" 2018-09-26T17:32:53,630 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Cache-Control: private, must-revalidate, max-age=0[\r][\n]" 2018-09-26T17:32:53,630 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Content-Type: application/json;charset=UTF-8[\r][\n]" 2018-09-26T17:32:53,630 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "ETag: bd8f0abc[\r][\n]" 2018-09-26T17:32:53,630 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Vary: Origin[\r][\n]" 2018-09-26T17:32:53,631 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Server: Microsoft-IIS/10.0[\r][\n]" 2018-09-26T17:32:53,631 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Set-Cookie: AGS_ROLES=xxxxxxxx; Expires=Thu, 27-Sep-2018 00:33:53 GMT; Path=/server/rest; HttpOnly[\r][\n]" 2018-09-26T17:32:53,631 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Server: [\r][\n]" 2018-09-26T17:32:53,631 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "X-AspNet-Version: 4.0.30319[\r][\n]" 2018-09-26T17:32:53,631 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "X-Powered-By: ASP.NET[\r][\n]" 2018-09-26T17:32:53,631 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Date: Thu, 27 Sep 2018 00:32:53 GMT[\r][\n]" 2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Content-Length: 16[\r][\n]" 2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "[\r][\n]" 2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "{"success":true}" 2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << HTTP/1.1 200 OK 2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Cache-Control: private, must-revalidate, max-age=0 2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Content-Type: application/json;charset=UTF-8 2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << ETag: bd8f0abc 2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Vary: Origin 2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Server: Microsoft-IIS/10.0 2018-09-26T17:32:53,633 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Set-Cookie: AGS_ROLES=xxxxxxxx; Expires=Thu, 27-Sep-2018 00:33:53 GMT; Path=/server/rest; HttpOnly 2018-09-26T17:32:53,633 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Server: 2018-09-26T17:32:53,633 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << X-AspNet-Version: 4.0.30319 2018-09-26T17:32:53,633 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << X-Powered-By: ASP.NET 2018-09-26T17:32:53,633 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Date: Thu, 27 Sep 2018 00:32:53 GMT 2018-09-26T17:32:53,633 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Content-Length: 16 2018-09-26T17:32:53,633 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | MainClientExec                   | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connection can be kept alive indefinitely 2018-09-26T17:32:53,634 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | PoolingHttpClientConnectionManager | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connection [id: 43][route: {s}->https://your-machine.domain:443] can be kept alive indefinitely 2018-09-26T17:32:53,635 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | PoolingHttpClientConnectionManager | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connection released: [id: 43][route: {s}->https://your-machine.domain:443][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20] 2018-09-26T17:32:53,635 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | Http                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Got response from HTTP request: {"success":true}. 2018-09-26T17:32:53,635 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | FeatureServiceOutboundTransport  | 77 - com.esri.ges.framework.transport.featureservice-transport - 10.6.0 | Response was {"success":true}.‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ Keywords that you are searching for to identify the block of messages above are: /rest/services/ (your service's name) /FeatureServer/ (layer index) /deleteFeatures I chose to open the karaf.log file in a text editor (Notepad ++) and search the content using the regular expression where.*reported_dt to quickly locate lines which include the SQL expression for my feature record's date/time field name: In the large block of DEBUG messages above, on Line 4, as you scroll to the right, you'll find a parameter added to the HTTP request:  Adding parameter (where/reported_dt < timestamp '2018-09-27 00:30:53') Notice that the message was logged at 2018-09-26T17:32:53 but the query parameter's value is +7 hours (00:30:53) and two minutes earlier (00:30:53 rather than 00:32:53). Illustrating this differently, using the search results from Notepad ++, you should expect to see "Adding parameter" logged by the HTTP Client every 30 seconds as it prepares another request to send to the feature service's \deleteFeatures endppoint: The bulk of the DEBUG log messages in the listing above, Lines 5 - 71, are a worker thread building out the HTTP request. What you really care about from this section is on Line 45. Scrolling to the right, past the f=json and the token=xxxxxxxx you'll see an HTTP encoded WHERE clause which specifies the timestamp parameter: where=reported_dt+%3C+timestamp+%272018-09-27+00%3A30%3A53%27 The feature service is going to receive a request which contains this literal SQL. If this WHERE clause does not match what SQL Express is expecting (since that is the database you are using) the request is probably going to fail. The DEBUG messages on Line 75 and Line 76 show that the both the HTTP Client and the Feature Service outbound transport see the executed request returning {"success":true} for my test. You might see errors logged by ArcGIS Server, or by your database engine ... but I'm not sure how to illustrate those as I'm not able to reproduce the issue using the PostGRE geodatabase I've deployed as part of my ArcGIS Enterprise. If you are able to determine that the date/time values written into each feature record are correct, I would recommend perhaps changing your GeoEvent Server output's delete interval from 20 seconds to 300 seconds so the queries to delete features occur every five minutes. If you can capture DEBUG log messages which show incorrect values for the query parameters and SQL being generated by GeoEvent Server, please open an incident with Esri Tech Support so that they can work to reproduce the issue using SQL Express. Hope this information is helpful -- RJ ... View more

Braulio Galvez‌ - There was a certificate issue on the server geoeventsample1.esri.com which has been addressed. Are you able to discover the following services at the ArcGIS REST Services Directory endpoint https://geoeventsample1.esri.com:6443/arcgis/rest/services ? Services: FAAStream (StreamServer) LABus (StreamServer) NYCMonitoredVehicleJourney (StreamServer) SeattleBus (StreamServer) WashingtonMetroBuses (StreamServer) WorldSatellites (StreamServer) If you navigate to the stream service's subscription page (e.g. Home > services > LABus > subscribe) and then click the 'Subscribe' button, are you seeing feature records broadcast by the stream service? I've noticed that a few of the stream services, like the NYCMonitoredVehicleJourney, are not broadcasting data as frequently as others. So while it may look like the stream service is "dead", if you wait a minute or so feature records do show up in the stream service subscription page's scrolling text display eventually. The velocity for FAAStream, LABus, and WorldSatellites is pretty high. I'm seeing data broadcast immediately when I subscribe to any of those stream services. - RJ ... View more