Community

delete old features

1632
3
Jump to solution
09-25-2018 01:10 PM
BrianLomas
by
Occasional Contributor III
‎09-25-2018 01:10 PM

I would like to use the 'delete old features' setting on the 'Add a Feature' output in order to purge old records. I have a date field which is calculated using "receivedTime()". However, when I use the settings in the image below the records are not deleted. I have tried inputting unix time into a integer field (date field will not accept it) but I am unable to select anything but a date field for the "Time Field in Feature Class". Can someone give me some guidance on using the "Delete Old Features" setting for the "Add a Feature output"?

Using Geoevent 10.6, SQL Express database

Thanks 

GeoEvent‌

RJ Sunderman‌ do you have any ideas?

0 Kudos
1 Solution

Accepted Solutions
BrianLomas
by
Occasional Contributor III
‎10-12-2018 09:02 AM

An update on this issue...

We have recently changed from using the SQL Express to using a Postgres SQL. By doing so, the "Delete Old Features" ability in Geoevent is working. Everything is set the same as what it was when using the SQL Express database. So, there must be something in the way SQL Express stores or interrupts the time information. 

View solution in original post

0 Kudos
3 Replies
RJSunderman
by Esri Regular Contributor
Esri Regular Contributor
‎09-26-2018 07:55 PM

Hello Brian -

I do not have an ArcGIS Enterprise I can easily configure with SQL Express ... so I'll have to show you the debugging steps necessary to generate log messages which will (hopefully) tell us what the issue is. It may be that the query parameters generated by GeoEvent Server's output and passed with the HTTP request to the feature service are not compatible with the SQL Express database.

But before we go there, let's check to make sure the date/time values that are being written into the feature record in the database are value we expect.

To set-up this test I created a feature service whose schema matches the following GeoEvent Definition. Notice that my attribute field reported_dt is a Date. If the field is not a Date the Field Calculator will not write the correct value into the field when evaluating its expression receivedTime() and the GeoEvent Server's output will not be able to use the field when requesting features be deleted.

Here's the GeoEvent Service I configured to ingest data with a String/TrackID and build a Geometry from two Double values Latitude and Longitude:

Notice the Field Mapper is preparing my event structure to match the structure and attributes of feature records expected by the feature class. The field reported_dt is left unmapped because the Field Calculator is writing the value receivedTime() into that field. I could also have used the Field Mapper to map the event property $RECEIVED_TIME into that field and elected to not use the Field Calculator at all - but you indicated in your post that you wanted to use the Field Calculator's function to obtain the event record's received date/time.

At 5:32 (and 20 seconds) by my server's clock, I sent an event record to GeoEvent Server and observed the following feature record created in my feature class:

 "features": [
  {
   "attributes": {
    "oid": 401,
    "track_id": "VWY-86-ABU",
    "reported_dt": 1538094740000
   }
  }
 ]‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Note that the date/time recorded in the feature class is not my server's local time, it is an epoch long integer whose value is UTC / GMT. This is important because whatever value is passed as part of a deleteFeatures request will have to reflect the UTC time, not my server's local time. (The value 1538094740000 is 12:32:20 am the next day, seven hours ahead of my current local time, 5:32:20 pm...)

Your configuration of your output looks correct. For my test I chose to have the output delete any feature records which are older than two minutes, attempting to delete "old" records every 30 seconds:

If I turn DEBUG logging on for GeoEvent Server, every 30 seconds I should see a block of log messages similar to the following. Unfortunately the information we want to capture is logged as part of the HTTP Client ... and there is a lot of HTTP traffic logged when you turn on DEBUG logging for that component, so I'll try to bullet for  you the information from the log messages below that you need to look for:

 

2018-09-26T17:32:53,495 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | Http                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Adding parameter (f/json).
2018-09-26T17:32:53,495 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | Http                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Adding parameter (token/xxxxxxxx).
2018-09-26T17:32:53,495 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | Http                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Adding parameter (rollbackOnFailure/true).
2018-09-26T17:32:53,495 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | Http                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Adding parameter (where/reported_dt < timestamp '2018-09-27 00:30:53').
2018-09-26T17:32:53,498 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | Http                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Executing following request: POST https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures HTTP/1.1
2018-09-26T17:32:53,499 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | RequestAddCookies                | 52 - com.esri.ges.framework.httpclient - 10.6.0 | CookieSpec selected: default
2018-09-26T17:32:53,499 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | RequestAuthCache                 | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Auth cache not set in the context
2018-09-26T17:32:53,499 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | PoolingHttpClientConnectionManager | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connection request: [route: {s}->https://your-machine.domain:443][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
2018-09-26T17:32:53,500 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | PoolingHttpClientConnectionManager | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connection leased: [id: 43][route: {s}->https://your-machine.domain:443][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]
2018-09-26T17:32:53,500 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | MainClientExec                   | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Opening connection {s}->https://your-machine.domain:443
2018-09-26T17:32:53,500 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | DefaultHttpClientConnectionOperator | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connecting to YOUR_MACHINE.DOMAIN/10.27.102.67:443
2018-09-26T17:32:53,501 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connecting socket to YOUR_MACHINE.DOMAIN/10.27.102.67:443 with timeout 0
2018-09-26T17:32:53,502 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2]
2018-09-26T17:32:53,502 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, ...
2018-09-26T17:32:53,502 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Starting handshake
2018-09-26T17:32:53,513 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Secure session established
2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 |  negotiated protocol: TLSv1.2
2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 |  negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 |  peer principal: CN=your-machine.domain, OU=Business Development, O=Esri, L=Redlands, ST=California, C=US
2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 |  peer alternative names: [your-machine.domain]
2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | SSLConnectionSocketFactory       | 52 - com.esri.ges.framework.httpclient - 10.6.0 |  issuer principal: CN=ESRI Enterprise Root, DC=empty, DC=local
2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | DefaultHttpClientConnectionOperator | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connection established 10.27.102.67:60152<->10.27.102.67:443
2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | MainClientExec                   | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Executing request POST /server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures HTTP/1.1
2018-09-26T17:32:53,514 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | MainClientExec                   | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Target auth state: UNCHALLENGED
2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | MainClientExec                   | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Proxy auth state: UNCHALLENGED
2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> POST /server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures HTTP/1.1
2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> Content-Type: application/x-www-form-urlencoded
2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> charset: utf-8
2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> Referer: https://your-machine.domain:6143/geoevent/admin/datastores/agsconnection/default
2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> User-Agent: GeoEvent Server 10.6.0
2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> Content-Length: 314
2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> Host: your-machine.domain
2018-09-26T17:32:53,515 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> Connection: Keep-Alive
2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> Accept-Encoding: gzip,deflate
2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "POST /server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures HTTP/1.1[\r][\n]"
2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "Content-Type: application/x-www-form-urlencoded[\r][\n]"
2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "charset: utf-8[\r][\n]"
2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "Referer: https://your-machine.domain:6143/geoevent/admin/datastores/agsconnection/default[\r][\n]"
2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "User-Agent: GeoEvent Server 10.6.0[\r][\n]"
2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "Content-Length: 314[\r][\n]"
2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "Host: your-machine.domain[\r][\n]"
2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "Connection: Keep-Alive[\r][\n]"
2018-09-26T17:32:53,516 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "Accept-Encoding: gzip,deflate[\r][\n]"
2018-09-26T17:32:53,517 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "[\r][\n]"
2018-09-26T17:32:53,517 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 >> "f=json&token=xxxxxxxx&rollbackOnFailure=true&where=reported_dt+%3C+timestamp+%272018-09-27+00%3A30%3A53%27"
2018-09-26T17:32:53,630 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "HTTP/1.1 200 OK[\r][\n]"
2018-09-26T17:32:53,630 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Cache-Control: private, must-revalidate, max-age=0[\r][\n]"
2018-09-26T17:32:53,630 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Content-Type: application/json;charset=UTF-8[\r][\n]"
2018-09-26T17:32:53,630 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "ETag: bd8f0abc[\r][\n]"
2018-09-26T17:32:53,630 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Vary: Origin[\r][\n]"
2018-09-26T17:32:53,631 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Server: Microsoft-IIS/10.0[\r][\n]"
2018-09-26T17:32:53,631 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Set-Cookie: AGS_ROLES=xxxxxxxx; Expires=Thu, 27-Sep-2018 00:33:53 GMT; Path=/server/rest; HttpOnly[\r][\n]"
2018-09-26T17:32:53,631 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Server: [\r][\n]"
2018-09-26T17:32:53,631 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "X-AspNet-Version: 4.0.30319[\r][\n]"
2018-09-26T17:32:53,631 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "X-Powered-By: ASP.NET[\r][\n]"
2018-09-26T17:32:53,631 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Date: Thu, 27 Sep 2018 00:32:53 GMT[\r][\n]"
2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "Content-Length: 16[\r][\n]"
2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "[\r][\n]"
2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | wire                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << "{"success":true}"
2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << HTTP/1.1 200 OK
2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Cache-Control: private, must-revalidate, max-age=0
2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Content-Type: application/json;charset=UTF-8
2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << ETag: bd8f0abc
2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Vary: Origin
2018-09-26T17:32:53,632 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Server: Microsoft-IIS/10.0
2018-09-26T17:32:53,633 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Set-Cookie: AGS_ROLES=xxxxxxxx; Expires=Thu, 27-Sep-2018 00:33:53 GMT; Path=/server/rest; HttpOnly
2018-09-26T17:32:53,633 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Server: 
2018-09-26T17:32:53,633 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << X-AspNet-Version: 4.0.30319
2018-09-26T17:32:53,633 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << X-Powered-By: ASP.NET
2018-09-26T17:32:53,633 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Date: Thu, 27 Sep 2018 00:32:53 GMT
2018-09-26T17:32:53,633 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | headers                          | 52 - com.esri.ges.framework.httpclient - 10.6.0 | http-outgoing-43 << Content-Length: 16
2018-09-26T17:32:53,633 | DEBUG | HttpRequest Worker Thread: https://your-machine.domain/server/rest/services/SampleRecord-UpdateFeature/FeatureServer/0/deleteFeatures | MainClientExec                   | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connection can be kept alive indefinitely
2018-09-26T17:32:53,634 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | PoolingHttpClientConnectionManager | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connection [id: 43][route: {s}->https://your-machine.domain:443] can be kept alive indefinitely
2018-09-26T17:32:53,635 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | PoolingHttpClientConnectionManager | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Connection released: [id: 43][route: {s}->https://your-machine.domain:443][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]
2018-09-26T17:32:53,635 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | Http                             | 52 - com.esri.ges.framework.httpclient - 10.6.0 | Got response from HTTP request: {"success":true}.
2018-09-26T17:32:53,635 | DEBUG | OutboundFeatureServiceCleanerThread-[Default][/][SampleRecord-UpdateFeature][0][FeatureServer] | FeatureServiceOutboundTransport  | 77 - com.esri.ges.framework.transport.featureservice-transport - 10.6.0 | Response was {"success":true}.‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Keywords that you are searching for to identify the block of messages above are:

  • /rest/services/ (your service's name) /FeatureServer/ (layer index) /deleteFeatures

I chose to open the karaf.log file in a text editor (Notepad ++) and search the content using the regular expression where.*reported_dt to quickly locate lines which include the SQL expression for my feature record's date/time field name:

In the large block of DEBUG messages above, on Line 4, as you scroll to the right, you'll find a parameter added to the HTTP request:  Adding parameter (where/reported_dt < timestamp '2018-09-27 00:30:53')

Notice that the message was logged at 2018-09-26T17:32:53 but the query parameter's value is +7 hours (00:30:53) and two minutes earlier (00:30:53 rather than 00:32:53).

Illustrating this differently, using the search results from Notepad ++, you should expect to see "Adding parameter" logged by the HTTP Client every 30 seconds as it prepares another request to send to the feature service's \deleteFeatures endppoint:

The bulk of the DEBUG log messages in the listing above, Lines 5 - 71, are a worker thread building out the HTTP request. What you really care about from this section is on Line 45. Scrolling to the right, past the f=json and the token=xxxxxxxx you'll see an HTTP encoded WHERE clause which specifies the timestamp parameter:

where=reported_dt+%3C+timestamp+%272018-09-27+00%3A30%3A53%27

The feature service is going to receive a request which contains this literal SQL. If this WHERE clause does not

match what SQL Express is expecting (since that is the database you are using) the request is probably going to fail.

The DEBUG messages on Line 75 and Line 76 show that the both the HTTP Client and the Feature Service outbound transport see the executed request returning {"success":true} for my test. You might see errors logged by ArcGIS Server, or by your database engine ... but I'm not sure how to illustrate those as I'm not able to reproduce the issue using the PostGRE geodatabase I've deployed as part of my ArcGIS Enterprise.

If you are able to determine that the date/time values written into each feature record are correct, I would recommend perhaps changing your GeoEvent Server output's delete interval from 20 seconds to 300 seconds so the queries to delete features occur every five minutes. If you can capture DEBUG log messages which show incorrect values for the query parameters and SQL being generated by GeoEvent Server, please open an incident with Esri Tech Support so that they can work to reproduce the issue using SQL Express.

Hope this information is helpful --

RJ

0 Kudos
BrianLomas
by
Occasional Contributor III
‎09-27-2018 07:18 AM

Hey RJ, 

Thanks for the thorough answer and for getting back with me so quickly, I appreciate it.

The 'received_time' field is not being mapped.

The field calculator for the 'received_time' field...

My 'received_time'  field is set to date in the GE definition 

A query on the Rest endpoint confirms the time is being processed as Epoch time. 

The log file appears to have the appropriate where phrase and the HTTP response appears to be successful. However, no records are being deleted from the feature class. The feature class records range from 9/23/2018 - current date/time. 

0 Kudos
BrianLomas
by
Occasional Contributor III
‎10-12-2018 09:02 AM

An update on this issue...

We have recently changed from using the SQL Express to using a Postgres SQL. By doing so, the "Delete Old Features" ability in Geoevent is working. Everything is set the same as what it was when using the SQL Express database. So, there must be something in the way SQL Express stores or interrupts the time information. 

0 Kudos