topic Re: Populate field from one hosted feature to another with GeoEvent in ArcGIS GeoEvent Server Questions

Populate field from one hosted feature to another with GeoEvent

JamieLambert — Mon, 20 Dec 2021 11:26:22 GMT

Hi All,

Don't have much experience with GeoEvent... I am trying to update a single field in one hosted feature with a field from another hosted feature using GeoEvent. The Input type is 'Poll an ArcGIS Server for Features' and the Output type is 'Update a Feature'. Both the input and output count go up when a record meets the filter on the input (Complete = 'Yes'), though the value is not written to my output feature. Both hosted features have a 'Job ID' field, and I have set the TRACK_ID to these features for the Input and Output Definitions – not sure if this is correct. I have used the Field Mapper to map the 'Complete' field from the Source to the 'Complete' field of the Target. This is the only value I am after.

I'm sure it is something simple I am missing or have not setting up correctly. Do I need to add a filter for Job_ID = Job_ID so the right Output feature gets updated? I have assumed TRACK_ID manages this. I'm hoping someone can point me in the right direction or share an example? Thanks.

Jamie.

Re: Populate field from one hosted feature to another with GeoEvent

JakeSkinner — Mon, 20 Dec 2021 13:43:31 GMT

Hi @JamieLambert,

Do you have the JOB_ID field set for the Unique Feature Identifier Field for the Update a Feature output? GeoEvent will use this field to determine if a new record should be created, or an existing record will be updated.

Also, are the geometries the same between the two feature services? If they are not, this may be causing the event to fail.

Re: Populate field from one hosted feature to another with GeoEvent

JamieLambert — Tue, 21 Dec 2021 06:10:05 GMT

Thanks @JakeSkinner.

A couple of updates after a meeting with the business line yesterday - 'job_id' is now replaced by 'sequence_id' as 'job_id' is not always unique. And 'Complete' is now 'ReportSent' (Input) and 'report_sent' (Output). The overall GeoEvent (10.8.1) setup is still the same and still not writing the value to the existing output record.

The Input is filtered to only records where a report has been sent (i.e. complete) and to Incremental Updates as once the report has been sent the record is no longer edited.

The Output Unique Feature Identifier is 'sequence_id'.

And 'sequence_id' is the TRACK_ID field for the Input Definition (I am assuming it is not required for the Output Definition as Output Unique Feature Identifier is already set).

Field Mapper in the Service maps the Input 'ReportSent' field to the Output 'report_sent' field.

The geometries are the same - both polygon.

I'm assuming that with the 'sequence_id' as Unique Feature Identifier and TRACK_ID assigned to the Input this will match the Input and Output records, then the Field Mapper will populate the Output 'report_sent' field with the Input value 'ReportSent'.

I think I have captured everything in the image below:

The 'ReportSent' value is being mapped, though it seems like the corresponding 'sequence_id' record in the Output hosted feature is not being picked up / written to. The Output feature is editable. Do I need to add a Filter or some other process? Thanks.

Jamie.

Re: Populate field from one hosted feature to another with GeoEvent

JakeSkinner — Tue, 21 Dec 2021 11:49:11 GMT

In your Field Mapper, include the sequence_id fields. These will be needed so it knows which features to update.

Re: Populate field from one hosted feature to another with GeoEvent

JamieLambert — Tue, 21 Dec 2021 12:34:00 GMT

The sequence_id is already in both hosted features. I thought the Field Mapper was to copy a field from the Input and write it to the Output?

Or is this how the Unique Feature Identifier setting works? You assign that value then use Field Mapper to tie the Input and Output records based on how that value is mapped?

Happy to give it a crack anyway!

Thanks.

Re: Populate field from one hosted feature to another with GeoEvent

JakeSkinner — Tue, 21 Dec 2021 13:09:05 GMT

In your field mapper, you're only sending the field to update. It does not know which record to update without sending the sequence_id.

Re: Populate field from one hosted feature to another with GeoEvent

RJSunderman — Tue, 21 Dec 2021 20:56:17 GMT

@JamieLambert -- Have you taken a look at the blog Debug Techniques - Add/Update Feature Outputs? If you request DEBUG logging on the com.esri.ges.transport.featureService.FeatureServiceOutboundTransport component logger you can see the REST requests your configured Update a Feature output is sending to the ArcGIS Server feature service. The DEBUG logs in GeoEvent Server give a lot of information you can use to figure out what's missing from the REST requests. The logging was reworked for the 10.8.1 release with even more information provided via TRACE level logging, if you need the component logger to be most verbose.

You can also work with the updateFeatures endpoint for the feature service in the ArcGIS REST Services Directory (taking GeoEvent Server out of the equation for now, while debugging) to iterate and try different requests to see what a web client needs to send to be able to successfully update feature records.

Re: Populate field from one hosted feature to another with GeoEvent

JamieLambert — Thu, 23 Dec 2021 08:27:35 GMT

If I only have sequence id and report sent in the Input and Output Definitions, and map these two fields, nothing happens.

With an Input and Output Definition with all fields, though only sequence id and report sent mapped, those two fields are updated in the Output hosted feature and all other fields are overwritten to null.

I don't understand why fields not mapped are being updated. Is it even possible to update one field and leave the rest untouched? Is the Field Enricher (Feature Service) processor required?

Thanks.

Re: Populate field from one hosted feature to another with GeoEvent

JamieLambert — Thu, 23 Dec 2021 01:28:43 GMT

Thanks @RJSunderman. I'll take a look at this, though I really expected this workflow to be pretty straight forward...

Jamie.

Re: Populate field from one hosted feature to another with GeoEvent

JakeSkinner — Thu, 23 Dec 2021 13:54:07 GMT

Try with the attached datasets. Publish each as editable feature services (i.e. Airports_Source, Airports_Target). In the below example, I published the services as hosted ArcGIS Server services. Here is how I set up GeoEvent:

1. Create a Poll an ArcGIS Server For Features input for the Airports_Source feature service

2. Create a Update a Feature output for the Airports_Target using name field as the Unique Feature Identifier Field:

3. Create a new GeoEvent Definition with only the name and fcc fields:

4. Create a GeoEvent Service. Add a Field Mapper Processer between the input and output:

I did not have to specify any fields as the TRACK_ID in this example. Let me know if you are able to get this to work on your end.

Re: Populate field from one hosted feature to another with GeoEvent

JamieLambert — Fri, 24 Dec 2021 06:08:08 GMT

I've marked "In your Field Mapper, include the sequence_id fields. These will be needed so it knows which features to update" as the solution as I was missing this part, as well as the sample provided thats helped me validate my configuration.

I think the rest of my setup was fine, though I think it was jammed up with my output definition that I was playing around with while I was trying to make it work as i wanted (missing the sequence_id in the Field Mapper obviously didn't help!).

Using the example dataset provided below (Sample.gdb.zip) and creating the service as outlined, then adding complexity one step at a time (input filter, update only, etc.) I was able to validate those steps successfully.

Once I was happy the setup was correct, I deleted everything from my original service (input, output, definitions, service) and rebuild from scratch, and no issues! A frustrating lesson, though a good result.

Thanks for all the help @JakeSkinner!

Re: Populate field from one hosted feature to another with GeoEvent

RJSunderman — Thu, 30 Dec 2021 16:15:53 GMT

Hey Jamie --

... those two fields are updated in the hosted feature [but] all other fields are overwritten to null. I don't understand why fields not mapped are being updated. Is it even possible to update one field and leave the rest untouched? Is the Field Enricher (Feature Service) processor required?

I'm not sure the question above was ever answered, so as a follow-up:

You are correct when you said, earlier, that the purpose of a Field Mapper is to copy data from one data structure / schema / GeoEvent Definition to another.
You generally use a Field Mapper to simplify and/or flatten a data structure to match the schema a Feature Service expects. That way, when an outbound connector (or "Output") deconstructs an event record to form a REST request, it can send the request to a feature service and the data in the client request will conform to the feature service's requirements.
It is possible to update a single feature record attribute (or just a few attributes) and leave others untouched. I would refer you to the article Using a partial GeoEvent Definition to update feature records for details on how to do this. Bottom line up front: If you leave fields unmapped in a Field Mapper configuration you are actually writing null values into the target data structure. So, yes, the null values end up being used to update feature record attributes overwriting whatever data the feature record had.
A Field Enricher (Feature Service) processor does something very different from a Field Mapper. The latter copies data from an event record to organize existing data in a different data structure. A Field Enricher performs an attribute join between an event data record and an external data record to incorporate data not present in an event record (enriching the event record with related data).

Hope some of this helps clarify tasks the processors you are configuring are actually doing.

-- RJ