Issue with a Time Field to Identify Incremental Updates when Publishing Survey in Survey123 to Portal 10.3

Question

Hi,

I am currently having an issue when I publish a survey built in Survey123 to Portal 10.3. By default, Survey123 will generate fields automatically like CreationDate, Creator, EditDate, and Editor, which are helpful in tracking when edits are made on the backend of surveys.

The EditDate is a useful field since I am able to use this field in the “Time Field to Identify Incremental Updates” option in GeoEvent Processor to track updates to my database (which is generated by Survey123) once a new survey has been created, or one is updated on the back-end.

Right now I am able to successful get this to work when I publish a survey from Survey123 to my ArcGIS online account. When my Input in GeoEvent Processor Manager is set to my ArcGIS Online account for the ArcGIS Server Connection option, I am able to select the EditDate field and use that as my Time Field option.

However, for some reason when I take the same survey and publish it to Portal 10.3 via Survey123, these automatic field names are changed to created_user, created_date, last_edited_user, and last_edited_date.

This ends up causing me an issue when I use the last_edited_date in GeoEvent Processor for the Time Field to get my incremental updates, it doesn’t work properly. I keep getting constant alerts for the same survey even though the last_edited_date is not being updated.

Is there a reason why when publishing to Portal vs ArcGIS online that those automatic fields would be generated differently? Is there a way to change this or a method to get this to work properly with GeoEvent Processor?

Thanks

MatthewThomas7 · Answer

When Survey123 publishes the hosted service, it's publishing a regular hosted feature service to ArcGIS Online or Portal for ArcGIS with editor tracking enabled.  This same setting can be applied to any hosted feature service by editing the item in either application.  When editor tracking is enabled, the additional 4 fields are appended to the schema of the feature service.

The differences you found in the editor tracking fields are present in any hosted feature service, not just those published from Survey123.  This has something to do with the backend of processes of ArcGIS Online and ArcGIS Server.  The hosted feature service's fields within Portal are similar to the fields that are created on a feature class in ArcMap when you enabled editor tracking.  ArcGIS Online is handles it differently.

Generally, it's just a matter of different schemas between otherwise similar services, and shouldn't be problematic in most cases.  For what you're experiencing with your GeoEvent input connector, it may just be a matter of using a different GeoEvent Definition.  In the Site tab in GeoEvent Manager, in the GeoEvent Definitions tab, choose 'Import' and import a Definition from the hosted feature service within Portal.  You may need to replace the existing Definition if it was created using the same method from ArcGIS Online.  Reference the new Definition in the fs-poll-in input connector settings.

RJSunderman · Answer

Robert -

I cannot comment on why publishing your survey to Portal vs. ArcGIS Online would produce the differences in schema you are observing.

If you want to try and dig into why GeoEvent is receiving event records from Portal which have not been recently updated (when polling an ArcGIS Online feature service appears to work), you will need to configure GeoEvent to include DEBUG messages in its log. You might try setting DEBUG logging on just the following loggers within GeoEvent Manager.  If you see errors or warnings associated with these, please let us know.

* com.esri.ges.datastore.agsconnection.DefaultAGOLConnection
* com.esri.ges.datastore.agsconnection.DefaultArcGISServerConnection
* com.esri.ges.datastore.agsconnection.DefaultPortalConnection

Unfortunately the messages you need which include the WHERE clause GeoEvent uses to identify event records which have been recently added or changed is provided by HTTP client log messages. I say this is unfortunate because that particular component is very, very chatty. You can only imagine as GeoEvent is communicating via HTTP / REST with all sorts of stuff ...

If you set DEBUG logging on the GeoEvent <root> component -- which you can do for a short period of time, just long enough to catch your input polling and receiving newly updated records -- you can catch the HTTP client log messages and review exactly what GeoEvent is requesting from the feature service it is polling. You will want open the karaf.log in a text editor and search for a logged message with the following substrings:

* Executing following request: GET http://services.arcgis.com
* arcgis/rest/services/ your_feature_service_name /FeatureServer/ layer_index /query
* where=1%3D1+and+LastUpdated+%3E%3D+%272016-05-10+21%3A34%3A07

(decoded:  where=1=1 and LastUpdated >= '2016-05-10 21:34:07)

That last message is URL encoded and can be difficult to read, but that's how it's going to appear in the log file. You can copy/paste the URL encoded string into an online decoder (I like using the URL Decoder/Encoder​) to get the decoded string I included above. Notice that the decoded string contains a date/time allowing GeoEvent to specify that any feature returned should have a date/time greater than that value.

Crawling through the karaf.log is going to be a chore, but since you know the name of the feature service being polled and the name of the attribute field suppling the date/time value you should be able to zero in on messages of interest fairly quickly (click the illustrations below to open the thumbnails):

If you filter your karaf.log down to just messages with "Executing following request" and "Got response from HTTP request" you should start to see a pattern, at least in the ArcGIS Online example.

* This pair of messages is requesting information on the feature service:

* This pair is telling GeoEvent that there are no features which satisfy the query:

* Eventually, when there *are* features which satisfy the query, you should see messages like the following:

This third example is interesting because GeoEvent is following its query for features WHERE LastUpdated+%3E%3D+%271970-01-01+00%3A00%3A00.000%27 (e.g. LastUpdated >= '1970-01-01 00:00:00.000') with a query requesting only the OBJECTIDs be returned (e.g.  returnIdsOnly=true). GeoEvent has to do this so that it knows whether it got all of the features or whether a feature service is limiting the number of features returned in response to a client's query.

You might be asking "Why the date/time back in 1970??"   When an input which polls for incremental updates is first created a false date/time is incorporated in the query’s WHERE clause. This is because the new input has not had a chance to cache a date/time value and needs to make one up which will likely get everything the feature service has. Thereafter GeoEvent will substitute the greatest date/time value from the set of features returned in response to its query ... and the next query's WHERE clause will ask for only the features whose date/time is strictly greater-than the previous observed date/time.

Hopefully all that technical background and log file illustrations were helpful. You, and possibly whomever you work with in Esri Tech Support, will need that background on what GeoEvent is doing in order to diagnose why the query to the Portal item is not returning just the recently updated event records.

- RJ