Latest Contributions by RJSunderman

‎08-27-2020

Matej – Using my test data above, I can configure three MCFS (multicardinal field splitter) processors to split the data first on the group element metric_collection, then on the group element Level1_Group3, and finally on the array Group3_Items. Following this pattern I think you'll be able to achieve what you need to do. Above, I've illustrated the three event records routed from the final field splitter to the output. Please make sure you have downloaded the latest release of the field splitter processor bundle (Release 8 - June 24, 2020) for release 10.8.1 of ArcGIS. This version includes changes which allow multiple MCFS processors to be arrayed in series. If you are using an earlier release of ArcGIS, please use Release 7 (February 27, 2020). Hope this information is helpful – RJ

‎08-26-2020

Hello Matej – Unlike a Field Mapper or Field Calculator which needs you to specify the full "path" into a data structure, the Field Splitter and in particular Multicardinal Field Splitter processors only expect the name of the element at which you want the split to be applied. So the Field to Split parameter's value should only specify the element's base name, linearWithinLinearGNElement, not the full path to the element (e.g. groupOfLocations.globalNetworkLinear.linearWithinLinearGNElement). That said, you may have found a limit of what the processor is able to handle in terms of a deeply nested, hierarchical, multicardinal, data structure. To test what you are trying to do I created some sample JSON data which adapts using the GeoEvent Definition illustrated below. Note that it does not matter whether the data received is formatted as XML or JSON – the inbound connector has to adapt whatever data is received to create an event record which can be routed to a GeoEvent Service for processing. After receiving the JSON above and allowing the inbound connector to adapt it using the illustrated GeoEvent Definition, I can use a Field Mapper to extract, for example, the data value "Delta" (line 21 in the sample JSON) by specifying the full path down through the data structure to the "E301" element in the array: metric_collection.Level1_Group3.Group3_Items[1].E301 If I specify the same full path when configuring the Multicardinal Field Splitter, I see the error message you were able to capture from the system log: Field name <name> is invalid. However, if I specify only Group3_Items (the name of the array) as the field on which to conduct the split, I see a different error message in the log indicating a null pointer exception was encountered. I am in contact with one of our developers to review the source code of this processor. We will see what we can do to better handle the case where Group3_Items (the name of the array) is specified and why a null pointer is not being caught. The work around, for now, is to use multiple Multicardinal Field Splitter processors in series and "walk down" the hierarchy from the top to the multicardinal element on which you want the final split applied. I'll reply to this post with what I tested so you can see what I had to do to implement the work around. Please make sure you have downloaded the latest release of the field splitter processor bundle (Release 8 - June 24, 2020) for release 10.8.1 of ArcGIS. This version includes changes which allow multiple MCFS processors to be arrayed in series. If you are using an earlier release of ArcGIS, please use Release 7 (February 27, 2020). Hope this information is helpful – RJ

‎08-18-2020

Hello Jitendrudu – When replies were written for the original thread Re: GeoEvent process Oracle/SQL connector I don't think the GeoEvent Manager supported publishing a feature service. The functional server role for licensing hadn’t been invented yet and GeoEvent Server was still being referred to as the GeoEvent Processor Extension for ArcGIS Server. In recent releases, however, you can use GeoEvent Manager to publish Hosted Feature Layers to your Enterprise portal or map/feature services to a stand-alone ArcGIS Server. So on the one hand, nothing has changed with regard to the interface between GeoEvent Server and an RDMBS table. The interface is still a feature service since GeoEvent Server can only make REST requests against an Esri ArcGIS Server feature service to add or update feature records. But on the other, you can now publish feature services which do not necessarily have Geometry. A feature service is typically published using a client, like ArcMap, which will not allow you to publish a feature service without a geometry. ArcMap defines a feature class as a collection of feature records and a feature record is a data structure with both geometry and attributes describing some object in the real-world. GeoEvent doesn’t make the same assumptions regarding Geometry that ArcMap does though. Real-time data streaming from a sensor whose position does not change wouldn’t necessarily include a geometry or coordinate values with its data. GeoEvent Server doesn’t assume that all data records will have geometry. To GeoEvent Server, “geometry” is just another type of attribute like a String or a Date. Say, for example, you were to create a GeoEvent Definition with two attribute fields – one a Date the other a String – and didn’t include a Geometry field type. You can use any release of GeoEvent after 10.6.1 to publish a feature service to the ArcGIS Server beneath which GeoEvent Server is running. Without a Geometry, to any other client, the feature class's table looks like a non-spatial table. But ArcGIS Server provides a feature service interface to the table allowing you to write the Date and String data you want to it without having a Geometry. You would still configure an Update a Feature output, and made sure to use a Field Mapper to map whatever schema your inbound connector uses to adapt data you receive to the feature service’s schema (PostGRE, for example, insists on using only lower-case field names; Oracle only upper-case field names). You shouldn't have any problem ingesting some sample data that has no geometry and using GeoEvent Server to add/update “feature records” via the feature service. The data records in the RDBMS simply have no geometry … so you won’t be able to add them to a web map as a feature layer for example, but I can query the data at REST. I assume if I had some other database client able to query from the data table I’d be able to retrieve the data that way, rather than going through the feature service’s REST interface to query the data. This would be a lot easier that developing a custom outbound transport which understood how to connect to the RDBMS using ODBC. This is possible, but I don’t know how feasible it is, really, given the inter-dependency of bundles in the Java system framework that underlies GeoEvent Server. GeoEvent Server’s whole design, in this case, assumes that you’ll be able to work through a feature service to access feature records. Hope this information is helpful – RJ

‎08-18-2020

Hello Rene -- You are correct. Documentation newly updated with the 10.8.1 release, particularly the on-line help topics Load Balancing and strategies for scalability, reliability, and resiliency, no longer discuss what I have referred to as a 'site' deployment approach (where multiple ArcGIS Server instances, each with a GeoEvent Server, are organized in a single ArcGIS Server site). Please take a look at the newly prepared on-line documentation; we've added quite a bit and have plans for more by the end of the year. Moving forward, the product team is recommending that all GeoEvent Server deployments follow a 'silo' deployment pattern. The distinction comes when you decide whether you have to deploy the same configuration to each machine, so you can use an external mechanism to route or distribute event records to two otherwise identical instances in parallel, or whether you can deploy a different configuration to each machine (allowing one instance to ingest event records of one type and another instance to handle event records of some other type). Update: February 2021 Beginning with the 10.9 release, every instance of GeoEvent Server you deploy must run beneath its own ArcGIS Server with its own ArcGIS Server site. This extends the single-machine high-availability active/active and single-machine high-availability active/passive deployment patterns promoted by ArcGIS Server. You will still be able to deploy multiple instances of GeoEvent Server which run independently from one another. These do not share a common configuration which GeoEvent Gateway must synchronize across a “cluster” of GeoEvent Server instances. (Recall that GeoEvent Gateway encapsulates the Apache Kafka message handler and the Zookeeper distributed configuration store used by GeoEvent Server.) We made the decision to remove support for multiple-machine / single-site deployments because we have found over time that GeoEvent Server deployments which coordinate through a single ArcGIS Server site do not meet reliability objectives. In rare cases of complete hardware failure -- where a single server node in a multi-machine deployment went permanently offline -- the deployment pattern we have deprecated did provide fault-tolerance. More frequently, however, when a deployment was challenged by a disadvantaged network, or a machine was temporarily unavailable, or servers were restarted out of sync, the whole deployment could become effectively unusable. System recovery for multitiple-machine / single-site deployments was tedious and error prone, which led to promises that a system architecture would provide high-availability failing to meet expectations. We therefore refactored the GeoEvent Gateway to provide better resiliency and overall system stability for the majority of our users by removing cluster leader election and in-sync replication between peer brokers/consumers. This means that multiple instances of GeoEvent Server (beginning with the 10.9 release) will no longer be able to synchronize a shared configuration or support a "clustered computing" architecture. But in the end, we achieve a better, more resilient, and more stable product. If you would like an informal write-up I prepared looking at some concerns surrounding this, please e-mail me directly: rsunderman .at. esri.com Hope this information is helpful, RJ

‎08-18-2020

Hello nasser hinai – I think what you want to do is use an Incident Detector with an opening condition which determines an asset has "entered" and area of interest (registered as a geofence) and set the closing condition to evaluate true when the asset's location has "exited" the geofence. The event record output from the Incident Detector will be an incident which has a very different schema from the event record you routed to the processor. While you'll lose a bunch of attributes from the original record, you will retain whatever field you tagged TRACK_ID and also GEOMETRY. The incident event record will also have a duration which you can pass through a filter. If the asset were to dwell within an area of interest for some time you could use a filter to check that: (a) the incident's status is 'Ended' (b) the incident's duration is greater than some value. Then whatever event records the filter allows to pass you send to an output configured to send an e-mail notification. Hope this information is helpful – RJ

‎08-18-2020

Hello Jose – To be clear, you are developing a custom web application which complements or reproduces what you can do using the GeoEvent Manager? So you're authenticating with the GeoEvent Server's administrative REST API to (probably automate) the creation of a bunch of processors, publishing a workflow connecting these processors as a GeoEvent Service. Does the service publish, albeit in an error state? If the service publishes you can export an XML representation of the configuration and inspect that to look for where the GeoEvent Service publication and validation is detecting a loop. There is no hard limit I'm aware of for how many processors you can place in a GeoEvent Service or how many different event flow connections you bring into or extent out from a node (e.g a processor or filter). There are certainly practical limits, and you should understand that every time a node handles an event there is some non-zero latency associated with that operation. Complicated real-time analytics applied through complicated event processing workflows through GeoEvent Services can severely impact the overall event records you can expect a GeoEvent Server instance to be able to handle. I have attached a simple illustration of an XML snapshot from an export of a GeoEvent Server configuration. Reading the XML, you'll be able to review the <flow> of a GeoEvent Service and see the <from> and <to> connections between every node. Hopefully the illustration helps with an audit / inspection of your more complicated GeoEvent Service. – RJ

‎08-06-2020

Hello Geoffrey West – Looking at the JSON you illustrated it looks to me like the data object is an array of JSON objects: "data": [ { "id": "b2c2626d-ff0b-4892-8262-5acae60946a4", "name": "JUPITER II", "mmsi": 701000533, ...‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ You probably need to add an index into your hierarchy parsing string to change data.last_known_position.geometry.coordinates[0] to data[0].last_known_position.geometry.coordinates[0] Also, I noticed that all of the negative double values seem to have an extra space between the '-' and the actual value. Quick experiments with GeoEvent Server seem to show that the extra white-space is being removed by the library used to parse the data ... I noticed https://jsonlint.com does the same thing ... but a different validator I often use triggered on these values and refused to validate the JSON to show it in a tree with collapsible nodes. Hope this information is helpful – RJ

‎08-05-2020

Hello Nicholas Gray – A couple of things you might try, to see if you're able to get a client to successfully subscribe to the stream service: On your mobile device, browse to https://websocket.org/echo.html For the Location enter wss://my-machine:6143/arcgis/ws/services/my-stream-service/StreamServer/subscribe Substitute your fully-qualified machine's domain name for my-machine and the name of your published stream service in the ArcGIS REST Services Directory for my-stream-service. You can find the basis for this URI if you click the JSON link in the top-left corner of the stream service's web page in the ArcGIS REST Services Directory. Scroll to the bottom of the JSON service spec to see the URI. I can browse to the ArcGIS REST Services Directory on my laptop and subscribe to the stream service via REST, and also use the WebSocket.org echo test to connect to see the feature record data as it is broadcast – probably because I've configured SSL certificates with trust for the Esri domain on the target machine and have a VPN running on the laptop enabling me to reach the machine on which GeoEvent Server is running. I found, for example, that I cannot browse to the ArcGIS REST Services Directory from my mobile. That would make trying to use the WebSocket.org echo test to connect to the stream service's web socket a non-starter. I can create a web map, again on my laptop, adding the stream service as a stream layer, and save it to my ArcGIS Online organization. I can log-in to ArcGIS Online from my mobile and access the web map – but I don't see any of the stream layer's data ... again, likely because without a VPN the web map client cannot access the machine running ArcGIS Server to browse its REST Services Directory or reach the web socket endpoint hosted by GeoEvent Server. Assuming that access to the machine from your mobile is not the problem, you might check the ArcGIS Server's SSL certificate configuration. The web browser on your mobile may need to see something to validate the HTTPS (or in this case WSS) connection. Using the WebSocket.org echo test is a good resource to see if clients external to your ArcGIS Enterprise are able to connect to a web socket being run by the GeoEvent Server. Hope this information is helpful – RJ

‎07-10-2020

Questions and Answers from the Expo Floor: What is ArcGIS GeoEvent Server GeoEvent Sever is one of several license roles for ArcGIS Enterprise. You license and install GeoEvent Server to add real-time capabilities to your Enterprise. GeoEvent Server, out-of-the-box, provides configurable inputs for connecting to a variety of data streams from virtually any data provider. What is a GeoEvent Service A GeoEvent Service is an event processing workflow you design and publish using the GeoEvent Manager web application. You drag, drop, and configure an input to specify how data will be received. You add and configure an output to specify what will be done with the data the GeoEvent Service processes. You then add one or more filters and processors to the service, connect the nodes to define an event processing workflow, and publish the GeoEvent Service. See Also: GeoEvent Server inputs GeoEvent Server filters GeoEvent Server processors GeoEvent Server outputs What's New in GeoEvent Server 10.8 / 10.8.1 Significant work went into simplifying common user workflows enabling users to maintain their focus on configuring and publishing GeoEvent Services. Most of what you need to do is now done from within the Service Designer rather than switching to a different web page in the GeoEvent Manager. The status, catalog, and option to create, edit, and delete primary service components (inputs, outputs, etc.) is now available from a consolidated Monitor page in the GeoEvent Manager. There is also a new GeoEvent Sampler utility built into the Service Designer allowing you to see the effect filters and processors you configure have on event records as they are received in real-time. For details, please see the PDF attached to this article. Question Answer Question Answer Question Answer

‎04-22-2020

Hello Mariela Del Rio – When you say automatic handling of "variable number of points in the routes", I'm not sure I follow what you are asking for. Looking over the GeoEvent Definition illustrations in the thread I don't see an attribute for a route or routes that is multicardinal (indicating the attribute is an array containing more than one value). Sara's reply to you on 10-February has a GeoEvent Definition whose event_time attribute looks kind of like a geoJSON geometry structure. The geoJSON specification for geometries places coordinates in an array. Our Generic-JSON inbound adapter includes a parameter asGeoJSON which can be used to tell the adapter to try parsing a field whose declared type is Geometry as a geoJSON geometry. The rest of the JSON structure is still handled as generic JSON, but any supposed geometry will be parsed differently, specifically so arrays of coordinate values can be interpreted as polylines or polygons. I'd have to experiment some with actual data from these feeds, but if this is what you're referring to we might be able to make something work for you without writing any code. If, on the other hand, you are looking for GeoEvent Server's inbound adapter to iterate over a variable number of JSON objects or discrete data values in an array ... that isn't something we have included in our product road map. GeoEvent Server's support for hierarchy and multicardinality is actually pretty limited and we drew the line quite some time ago at providing some sort of configurable iterator to iterate over a variable list of values in order to perform processing on each value. There are some really powerful things you can do using regular expressions and a series of Field Calculator processors. You might check out this thread for an example. Switching to your request for pagination for web service service(s), this is a pretty broad topic. There is no standard I'm aware of for how a given external web service might elect to communicate to a client that data retrieval should be performed as a series of queries rather than receiving all of the data as part of a single response to a single query. For example, the ArcGIS REST API for feature services specifies that a client should interpret a feature service returning exceededTransferLimit=true to mean that "there are more query results" and "you should continue to page through the results". GeoEvent Server is able to page through Esri's feature services when querying for a large number of feature records, but I'm not sure how we would implement a general solution for paging through any external web service's content. (I'm actually more familiar with the opposite, when a web service wants to return tens-of-megabytes of data to a client in response to a query and GeoEvent administrators ask how to configure GeoEvent Server to handle such a massive slug of data.) Your third ask, for automatic token renewal, is part of our product backlog. What we want to do is retire, at least, the obsolete "OAuth1" authentication support offered by inbound transports and implement new support for OAuth2 authentication, which I think would include automatic token renewal. Our last research spike into this showed that while OAuth2 is a standard, there are subtle ways a service author can define how the different authorization types are implemented. We can probably develop a connector which supports the OKTA standard of OAuth2, but the effort was put on hold. When discussing this problem with customers I tell them that this is one of the cleanest examples for recommending a custom transport that I can think of. There are a bunch of different ways a web service might ask a client to first authenticate and acquire a token so that a second request (with the token) can be made authorizing a client's request. I don't recall any of my team helping to develop the custom inbound transport (or processor?) you refer to, but you are correct, if the development was completed using the 10.6 version of the GeoEvent Server Java SDK, you should be safe to continue using it for 10.7.x and 10.8.x ... we've no plans for breaking changes to the API which would require the custom component you had Esri build for you to need to be re-compiled. Hope this information is helpful – RJ

‎03-25-2020

Very nicely presented Jake Skinner – I like the RegEx you used in the Field Calculator (Regular Expression). Your pattern match – (,[^,]{0}) – is very tight and well crafted. I do not use that variant of field calculator as often as I use the "regular" Field Calculator with its replaceAll( ) function, which changes how I think about regular expression pattern matching. A regular Field Calculator also allows me to combine operations in a single expression. I tried combining the string replacement and length operations with a slightly different RegEx. I'm not sure that all of the Field Calculator's supported functions can be combined this way, but it appears to work in this case: length(replaceAll(DataString, '[0-9]{1,}[,]*', 'x')) The above RegEx matches one-or-more digits (zero to nine) with a trailing comma (if one exists) and accepts a final numeric identifier without a comma as the last in the delimited list. Each identifier is replaced with a single literal 'x' so that the length( ) function can simply count the number of x characters. Seems to work for empty string (""), single identifier ("12345"), and multiple identifiers ("689102,0256,651,74290") ... returning lengths of 0, 1, and 4 respectively. Null strings are a special case; tests I ran suggest that if GeoTagger found no points in a precinct's polygon and chose to assign a NULL value to the GeoTags field rather than an empty string (I can never remember which value it chooses), the combined expression will produce a NULL value, So your final filter – rather than checking count > 0 – could check NOT(count ISNULL) allowing you to also update districts with a count of zero. This might be important if a precinct was previously updated with n>0 crimes and things got quiet so the crimes count needed to be reduced back to zero. Also, to cross-link to another topic/trick, if you do not want to carry the burden of mapping a bunch of fields whose values you do not intend to change (command_code, patrol_boro, sq_miles, category, etc.) you could use a Partial GeoEvent Definition with only two fields to update the count for each precinct feature record. Assuming precinct is your event record's TRACK_ID and unique feature identifier field, a GeoEvent Definition with only two fields (precinct and count) would protect against accidental changes to other fields by using precinct to identify which feature record to update and then only update the count for that precinct. Thanks for the write-up – RJ

‎03-24-2020

I agree with Eric Ironside, the problem described isn't a good fit for GeoEvent Server because every event record is generally considered atomic. I can't compute a sum or average value across several events without developing a custom processor to collect event records over a period of time and periodically dump a cache to compute a statistic. Taking a closer look at Simon Jackson initial ideas, we could import the catchment polygons as geofences and then ingest and process the river gauge point features to get the name of the catchment AOI the points fall within (using a GeoTagger). But that only sets us up for some sort of post processing operation external to GeoEvent Server; we've only used GeoEvent Server to enrich a feature record set of river gauges with an attribute identifying a catchment. Assuming neither the location or geometry of catchment polygons / river gauge points frequently change, this becomes an infrequent batch operation, not a recurring real-time operation. Turning the problem inside out, we could import the river gauge points as geofences then ingest and process the catchment polygons. A GeoTagger could be used to enrich each polygon with a comma separated list of sensor identifiers. But again, we only get identifying names from GeoTagger – if we split the delimited list in order to use a Field Enricher to pull actual flow rates into each record we're no better off that having ingested the point records in the first place ... we cannot rejoin the split delimited list to obtain metrics from several event records to compute an average. I suppose we could use a Field Calculator to append each river gauge's value to the gauge's name and use a stream service to broadcast these so that a geofence synchronization rule could actively update a set of geofences whose name provides both a sensorID and an observedValue ... then, as part of a different GeoEvent Service, ingest and process the catchment polygons using a GeoTagger to collect all the "enhanced" geofence names into a comma delimited list. A regular expression, like Adam Repsher suggests, could then (maybe) pull apart the delimited list, extract each sensor's observed value from the "geofence name" and compute an average ... But this is hardly the elegant solution Simon is looking for. It's forced, brutish, and fragile as we now have two independent GeoEvent Services polling catchments and river gauges, updating geofences, and presenting us with inherent race conditions. If you really wanted to do this using GeoEvent Server, you would want to develop a custom processor that incorporated a timer. The processor would catch and hold a series of GeoTagged points as long as its timer had not expired. As the processor was collecting data, it could enter metrics from each river gauge into some sort of key/value structure in memory using the catchment identifier as the key. The processor's timer would reset as new gauge event records are received and its in-memory data structure updated. If the timer ever expired ... say, 30 seconds after not seeing any new data ... it would compute averages for all the catchments it had key values for and write-out updates to the catchment feature records. The burden here is that you have to use the GeoEvent Server's Java SDK to create a custom processor. I don't see anything elegant you can do with GeoEvent Server out-of-the-box to solve this problem. Hope this information is helpful – RJ

‎03-23-2020

Also consider that as you collect copies of data files you intend GeoEvent Server to read from, you probably want to configure the GeoEvent Server's input to delete files it has read after they have been processed. It is expected behavior at the 10.7.x release (and I think 10.6.x as well) that if you stop a running input used to read data from a file, any file(s) you have in the watched folder will be re-read when the input is restarted. Same goes for server machine reboot. You probably do not want GeoEvent to re-read files following a restart ... You will still need to append a date/time string to the file name, to ensure the external process used to copy files does not, at any point, try to lock or overwrite a file GeoEvent Server is reading. There is no way to coordinate GeoEvent Server's read (or file delete when done reading) with the external file copy process; filename uniqueness is how you work around that. - RJ

‎03-10-2020

Hello William – To add to what Brad‌ says above, there are two approaches to approaching a multi-machine GeoEvent Server deployment. The first I refer to as the 'site' approach, the other as the 'silo' approach. The ‘site’ approach deploys multiple ArcGIS Server instances, each with a GeoEvent Server, in a single ArcGIS Server site. The GeoEvent Gateway is utilized more heavily in this configuration as the component responsible for event record distribution across the site. The ‘silo’ approach relies on an external broker or load balancing component such as Apache Kafka for message distribution. In this approach you are essentially configuring multiple independent GeoEvent Server instances and taking on the challenge of routing a portion of the inbound data you need to process to different instances. Brad mentioned one tutorial, GeoEvent Server 10.6.x Multiple-Machine Site, which covers the 'site' approach. There is another tutorial, GeoEvent Server Resiliency, which covers the 'silo' approach. There are pros and cons to both the 'site' and 'silo' approach. When choosing one over the other you should carefully consider your specific objectives – resiliency, scalability, fault-tolerance, reliability. Architects need to decouple these specific objectives from a more generic "high-availability" objective. Brad is correct that with the introduction of the GeoEvent Gateway in the 10.6.x release architects have the option to follow a 'site' deployment and allow GeoEvent Server to handle machine fail-over when a machine participating in a site fails. We've found on the product team, however, that when a multiple machine approach is necessary, accepting the technical debt of learning how to deploy, configure, and administer Apache Kafka and Zookeeper and following a 'silo' deployment model gives administrators better visibility into operational failures and more control over recovery. For this reason, more than any other, I am more comfortable recommending a 'silo' approach over the 'site' approach. I have written up some thoughts and advice on resiliency, scalability, reliability, high availability, and pros / cons to consider when taking on a multiple machine deployment. Brad or I can share this with you if you schedule some time with one of us to discuss your approach, concerns, and objectives. I will offer that, realistically, folks who are happy with GeoEvent Server are those who are able to get what they need out of a single machine deployment; folks who are unhappy with GeoEvent Server are those who try to architect solutions which push the technology on which GeoEvent Server was built beyond what it’s able to do by trying to design “highly available” solutions with multiple machines. Customers looking for more resilient solutions with auto-scaling and built-in fault-tolerance are encouraged to look at the new ArcGIS Analytics for IoT – a SaaS offering for ArcGIS Online. Its architecture and implementation are completely different from GeoEvent Server. You can read more about ArcGIS Analytics for IoT at the following links: ArcGIS Analytics for IoT Overview Blog: Introducing ArcGIS Analytics for IoT On-Line Help -- What is ArcGIS Analytics for IoT? Moving forward, the GeoEvent Server product team is not going to be recommending multiple machine deployments. We won't be taking away an architect's options to deploy using a 'site' or 'silo' approach, but we will be encouraging customers who have needs beyond what a single instance of GeoEvent Server can support to consider the ArcGIS Analytics for IoT SaaS offering over the on-premises ArcGIS GeoEvent Server. Hope this information is helpful – RJ

‎03-06-2020

I sent the following to one of our contractors today. The information on configuring SSL certificates, administrative tips for multi-machine deployments following a 'site' model, and things to check when GeoEvent Server fails to load its ArcGIS Server's configured certificates and instead uses its own SelfSignedCertificate might be of more general use, so I'll leave this here in case it helps someone working with GeoEvent Server deployments. With a multi-machine ‘site’ configuration it is critical that all machines trust one another. That means that not only do I have to configure an SSL certificate on Box#1 and configure that machine’s ArcGIS Server to use that certificate as its Web Server Certificate … I have to import certificates for Box#2, Box#3, … Box#N into the ArcGIS Server so that it trusts all the other machines participating in the site. I have to do this “fan-out” on every server, setting *that* server’s Web Server Certificate and importing certificates from all the *other* machines onto that server. I’ve captured what I do that works for me when setting up a couple of machines. But to be honest, SSL certificate configuration is not something I understand at a deep, technical level. Likely there is a “better” way of doing what I propose in the attached, maybe using a wild-card certificate, but I don’t know how to set that up. I’d also like to break the problem you’re seeing into two pieces. The first being SSL certificate configuration, for which I’ll capture some screenshots (see attached PDF). The second piece involves things I look at when GeoEvent Server seems unable to locate and load the certificates its ArcGIS Server is configured to use. The second part probably has more to do with why GeoEvent Server completes a fail over to use its SelfSignedCertificate rather than the certificate its ArcGIS Server is configured to use. I’ll apologize if anything I share is overly pedestrian … like I said, SSL certificates are not my cup of tea, so all I can do is show you what works for me and hope that your experience will allow you to iterate and adapt what I have to share. The first part, SSL certificate configuration, is attached. For the second part … I would caution against opening the Java Keystore using a command like keytool. I’ve watched developer’s do this, but I’ve never seen that administratively editing the JKS do anything to resolve a problem. GeoEvent Server, when it launches for the first time, interrogates its ArcGIS Server for information on its site and SSL certificates. If you would like to see some evidence for this, you can request DEBUG logging on the com.esri.ges.security.arcgis.sslconfig GeoEvent Server logger component. GeoEvent Server will attempt to copy the certificate configuration of the ArcGIS Server is it running beneath. If GeoEvent Server cannot obtain the certificates from the ArcGIS Server configuration, it will fail over to use its own SelfSignedCertificate. The fail over is intended to at least allow GeoEvent Server to complete its startup – but if GeoEvent Server does not trust machines the same way as its ArcGIS Server does, lots of stuff is probably not going to work. By the way, it is precisely because GeoEvent Server interrogates its ArcGIS Server for information that it is best to have your ArcGIS Enterprise (Portal for ArcGIS, hosting ArcGIS Server, ArcGIS Data Store) fully configured with a site created, federated and all SSL certificates configured before you introduce GeoEvent Server to the Enterprise. Installing – or at least starting the GeoEvent Gateway and GeoEvent Server – before ArcGIS Server and Portal for ArcGIS are fully configured means that the initial interrogation fails. Security topology may change … you may later decide to federate for example, or SSL certificates have to change … in which case resetting your GeoEvent Server configuration from within GeoEvent Manager (e.g. not an “administrative reset”) should force GeoEvent to pick-up changes made to the Enterprise configuration. Worst case you have to stop and restart GeoEvent Server after resetting its configuration then import your inputs, outputs, …etc. You don’t always have to re-install, but installation order can make your life easier administratively when deploying all this s/w for the first time. There are a few things I check when I find that GeoEvent Server is using its own SelfSignedCertificate rather than the certificate its ArcGIS Server specifies as its Web server SSL certificate. Did I accurately follow the certificate configuration laid out in the attached PDF? Sometimes a machine gets re-imaged, or a something else invalidates a certificate I had previously generated, applied, and imported using the attached procedure. That is when I have to walk through that whole process again. Sometimes it is just that a certificate has expired. They do that, and rarely when it’s convenient. ArcGIS Server maintains two different certificate stores – do their contents match? Seriously, this has bitten us more than once. There’s a certificate store beneath …\ArcGIS\Server\framework used, I think, by web clients. ArcGIS Server maintains a copy of these certificates in its configuration store for each machine in the site. This second key store is used, I think, by thick client applications. C:\Program Files\ArcGIS\Server\framework\etc\certificates C:\arcgisserver\config‑store\machines\MYMACHINE.ESRI.COM The two certificate stores should be identical. I’ve found once or twice that files had not been copied from the Server framework into its configuration store. When this happened I had to stop ArcGIS Server, manually create the folder named for the machine (e.g. CARMON.ESRI.COM beneath …\config-store\machines) and copy the files from the framework into the configuration store folder. When I restarted ArcGIS Server and administratively reset GeoEvent Server, it adopted its Server’s certificates and began working as expected. ArcGIS Server maintains both JSON and XML copies of its SSL configuration – do they match? When debugging we’ve found a couple of times that the SSL configuration reported by ArcGIS Server by its Admin API did not match an XML file’s content that GeoEvent Server was using to retrieve certificate information. Specifically a file D:\arcgisserver\config-store\machines\10.0.0.131.json specified a webServerCertificateAlias which did not match what should have been the same information in a C:\Program Files\ArcGIS\Server\framework\etc\machine-config.xml file. When this happens you might try stopping GeoEvent Server (and GeoEvent Gateway) and reconfiguring the ArcGIS Server’s certificates. If the files match after ArcGIS Server completes a restart, then you can administratively reset GeoEvent Server and it should pick-up the correct certificate configuration. Does the GeoEvent Gateway have its correct hostname / IP Address in its com.esri.ges.gateway.cfg file? Part of the GeoEvent Server administrative reset is to delete this file and make sure that it gets regenerated automatically when GeoEvent Gateway (or maybe its when GeoEvent Server) comes up for the “first” time. If you look at the file’s content in a text editor you’ll see that it instructs the Gateway as to which server and port it should use for connecting to the Zookeeper distributed configuration store which manages your GeoEvent Server’s configuration. It also specifies the Apache Kafka topic partitions, replication and how to reach the broker. If the machine information in this file designates a machine which does not exist – like when you use cloud image utilities to push a machine image out to multiple virtual machine instances – when GeoEvent Gateway launches it never reaches a stable state and cannot support its GeoEvent Server. The procedures to administratively reset GeoEvent Server are in a blog: Administratively Reset GeoEvent Server You can follow the procedures for 10.6.x as they will be the same for 10.7.x and 10.8 deployments. These are the steps, by the way that you have to run on each server when following a multi-machine deployment with a ‘site’ configuration and one of the machines drops out of the configuration and does not automatically re-integrate. Resetting a multi-machine ‘site’ configuration is both tedious and error prone. You basically have to work as if you’re installing all of the s/w for the first time: Install ArcGIS Server, create site, configure certificates, install GeoEvent Server Install ArcGIS Server, join site, configure certificates, install GeoEvent Server Install ArcGIS Server, join site, configure certificates, install GeoEvent Server (lather, rinse, repeat) When you already have an ArcGIS Server site with, say, three machines things get messy. I think what you do is use ArcGIS Server Manger to ‘STOP’ two of the machines – you’ll want to stop GeoEvent Gateway and GeoEvent Server on those machines first. The idea is that as far as the ArcGIS Server site is concerned it only has one machine. Complete the admin reset for GeoEvent Server on that machine then start its Gateway, wait a couple minutes, then start its GeoEvent Server. Then, back in ArcGIS Server Manger to ‘START’ a second machine. The site now thinks it has two machines, only one of which is running GeoEvent Server. Complete the admin reset for GeoEvent Server on the second machine then start its Gateway, wait a couple minutes, then start its GeoEvent Server. As the GeoEvent Gateway and GeoEvent Server come up they’ll discover and coordinate with the running GeoEvent Server, through the AGS site, and work out among themselves how to balance the kafka topics and brokers. Finally, in ArcGIS Server Manager, ‘START’ the third machine. The site now thinks it has thee machines, only two of which are running GeoEvent Server. Complete the admin reset for GeoEvent Server on the third machine then start its Gateway, wait a couple minutes, then start its GeoEvent Server. As the GeoEvent Gateway and GeoEvent Server come up on this final machine they’ll integrate with the other two. If you try to bring all three machines on-line at the same time and they were not properly integrated / balanced when they were taken down … they’ll likely not integrate correctly with one another. You have to stage their startup so that the ArcGIS Server site never has more than one machine ‘STARTED’ which does not have a fully initialized and integrated GeoEvent Server. When two or more GeoEvent Server’s try to integrate at the same time things tend to fail. It is precisely this sort of fragility, and the fact that it is so administratively difficult to determine if the machines were not properly integrated / balanced in the first place, that I feel a ‘site’ configuration really doesn’t provide the resiliency it was designed to provide. Sure, when everything is working it works beautifully. But when a machine falls out of configuration … getting the ‘site’ back to nominal is difficult (to say the least). Hope this information is helpful – RJ

Online Status	Offline
Date Last Visited	‎03-07-2024 07:12 PM

My Ideas

Latest Contributions by RJSunderman

Re: Question on Multicardinal Field Splitter Processor

Re: Question on Multicardinal Field Splitter Processor

Re: JDBC connector for Oracle/SQL connector

Re: GeoEvent Server High Availability

Re: Send email notification by calculating the time difference

Re: Loop Errors in GeoEvent

Re: GeoEvent Server - parsing many cardinality for array of coordinates to create geometry

Re: Issues w/ viewing a Stream Service on a Mobile Device

UC 2020 FAQ

Re: Poll an External Website for JSON using headers

Re: Summarize Point Count within Polygons using GeoEvent

Re: Update polygon with the average value of the points that are within it?

Re: CSV in a Shared Folder

Re: GeoEvent Server High Availability

Configuring SSL Certificates for a Machine Using Domain Enrollment

Re: GeoFence sync is not detecting deleted feature...

Re: GeoTab Connector with Multiple GeoTab Database...

Re: alert when 2 points are close to each others

Re: What time is it? Well That Depends...

Re: Shapefile as Input to GEP

ArcGIS Analyse