How to bulk insert multiple features from JSON array with Add Feature output connector in GeoEvent Server?

JamesMadden1

I have begun working with GeoEvent Server and for an early project I am using the Add Feature output connector to insert features from a standard JSON array.  The JSON is being piped in from the Receive JSON input connector.  I have noticed that GeoEvent Server will create an event for each element of the JSON array.  The JSON is structured so that each element represents a new feature.  I would like to know if there is a way to treat all of the array elements as a single event which in turn would lead to a single transaction.  GeoEvent Server seems to create the events very quickly but I can count the seconds that pass in between each new insert.  I can always call the Add Feature REST endpoint directly or run a stored procedure, but I would prefer to use GeoEvent Server if it will support the volume coming in.  Again, I have just begun working with the software and I imagine there might be a way to run a bulk insert.  The add feature output connector does have an option for setting a feature count transaction threshold, so there must be a way.  I may look at developing a custom transformation connector (this would be incredibly helpful wink wink) but I would prefer sticking with the out of the box tools for the time being.

I am not really sure how to resolve this issue.  There might be a solution that relates to the structure of the GeoEvent Definition or the structure of the JSON array.  Again, I would like to insert multiple rows in a single transaction using the add feature output connector.

Thanks in advance to any helpful responses.   

GregoryChristakos

Hi James,

To help clarify, are you looking to treat all objects in your JSON array as a single record (analogous to a single row in a database table), or are you looking to simply 'queue' individual records (objects in the array) in such a way that any adds/edits to your feature service is done in 'bulk'? You had asked:

I would like to know if there is a way to treat all of the array elements as a single event which in turn would lead to a single transaction.  

This would imply that you want to merge all array elements (or objects) so that they are a single event record (i.e. row in a database table).

Likewise you had also asked:

Again, I would like to insert multiple rows in a single transaction using the add feature output connector.

This would imply that you want to maintain each array element as its own respective event record. Rather then adding these one-by-one in real-time, you would like to add several in bulk. No?

I assume you are asking about how to achieve the latter scenario, but I would like to be completely sure since both questions entail entirely different responses. If I am to assume the latter, you might want to look into the Add a Feature output connector properties, "Update Interval (seconds)" and, as you've pointed out already, "Maximum Features Per Transaction".

The "Update Interval (seconds)" parameter is important here since it can be used to specify how frequently the output connector will essentially queue processed event records, make a request to the feature service of interest to add those processed event records as new feature records, and then finally flush its cache to make room for additional processed event records. By default, this property is set to 1 second but if you wanted to, you could adjust this to be something of your choosing like 2 minutes (120 seconds). Using 120 seconds as an example, GeoEvent Server will queue all processed event records (from your input connector) for 2 minutes. Once the 2 minute window has lapsed, GeoEvent Server will make a request to the feature service to add its queued event records as new feature records. GeoEvent Server will then flush its cache to make room for the next set of processed event records only to repeat the same process 120 seconds later. The important thing to keep in mind with this property is that it works using an iterating time interval, not record count (!= 'add every time 200 event records have been queued'). If you know that your data is coming into GeoEvent Server at a consistent rate (e.g. 20 event records per second), you could hypothetically adjust this property to match what you're looking for in respect to a certain 'transaction size'.

Of near equal importance here is the "Maximum Features Per Transaction" property. This property controls for how many records to include in any single feature service request to add new feature records. By default this is set to 500 which means that each add request will contain a maximum of 500 records. If we go back to the above example involving a 120 second update interval, let's say you've queued 1800 records over that period of time. This property will ensure that 4 transactions are made to the feature service (500 + 500 + 500 + 300 = 1800) to add new feature records. It's important to note that this property doesn't provide inhibit the number of requests made (!= 'only make 2 requests at the maximum transaction size regardless of how much data there is still'). You might be thinking that this property isn't very useful for your scenario, and that might be true, but I thought it is important to share in case you need the requests being made to your feature service to generally contain a certain record count (and nothing more). Realistically, this property was intended to help reduce the burden on an external server/service by providing transaction management.

If you're looking to find out more information regarding the former scenario, please feel free to let me know.