Latest Contributions by GregoryChristakos

Learn how to get started with consuming data from Spire's AirSafe Tracking Stream in ArcGIS Velocity using the HTTP Poller feed. ... View more

ArcGIS Velocity offers the ability to send an email as part of a configurable real-time or big data analytic. This is often useful for when there are events of interest that warrant the need for an immediate notification. For example, you may need to be notified when a vehicle in your fleet has experienced engine trouble and requires assistance right away. Likewise, it could be equally imperative that you are informed about summary statistics derived by your big data analytic after it has finished processing sensor data gathered over the last month. Either way, emails allow you to act upon your data that much faster.   The Email output in ArcGIS Velocity allows you to send an email via an externally accessible SMTP server. You can certainly configure your email output with your own organization’s SMTP server if it is externally accessible on the internet. However, if this is not an option, or you do not have an SMTP server of your own, it is worth considering alternatives such as what is offered by Twilio’s SendGrid SMTP service.  In this blog, we will explore how you can configure your email output to use SendGrid’s SMTP server as one option for sending emails in ArcGIS Velocity.   What is SendGrid and how can I use it?   SendGrid is a cloud-based SMTP service provider. A SendGrid account with the free Email API plan can enable you to send up to 100 emails a day via SendGrid’s SMTP server. Paid subscription tiers can support even more emails. But with a few simple steps, the SendGrid SMTP Server can be configured with ArcGIS Velocity to send emails as part of your real-time and big data analytics.   Before getting started, it is important to note that an email will be sent for each discrete feature that an instance of the Email output receives in ArcGIS Velocity. For example, if you have configured a feed that ingests 250 events per second, and this feed is used in a real-time analytic just to send email alerts, you could accidentally emit 250 emails every second without any sort of filtering of your data as a precursor. Talk about spam! This high rate of messaging could quickly overload the capacity of your organization’s SMTP server or result in service suspension from your SMTP service provider.   Consider the functional design of your analytic, as well as any rate limitations from external SMTP services, before sending emails via Velocity. It may be necessary to consider a paid SMTP service subscription to guarantee a large quantity or rate of emails. However, unless necessity dictates otherwise, the best practice with email outputs is to be conservative with the number of emails generated per second, minute, hour, and so forth.   Configure SendGrid for ArcGIS Velocity   To use SendGrid for your SMTP Server needs, you will first need a SendGrid account. If you do not have a SendGrid account already, you can create one here. You can configure an account using the free Email API to get started. Consider SendGrid’s terms of service before proceeding.   Once you have a SendGrid account, you will then want to create a single sender identity. A sender identity essentially is what represents you as your “from” email address. This is the email address that recipients will see as the sender and is what ArcGIS Velocity will use as the from email address. You can create a single sender identity from the SendGrid dashboard upon signing into your new SendGrid account. You will need to verify your single sender identity by checking the email you had specified. If you are using an email that is part of your organizations domain, you may want to consider reviewing any DMARC policies that are in place as this could interfere with using the email address as your single identity sender. In any case, follow the instructions provided by SendGrid to configure a single sender identity.   Create a sender identity after you sign into SendGrid After you specify a single sender identity, you should be ready to send your first email with SendGrid. There are two methods to send an email. The first is to integrate using SendGrid’s Web API or SMTP Relay and the second is to build and send using marketing campaigns. For ArcGIS Velocity, you will want to choose the first method so that you can configure an SMTP [server] relay.   Integrate using SendGrid's SMTP relay SendGrid will require that you first create an API key to authenticate with their API and send emails using their SMTP relay. The API Key acts as the password for sending emails, via the SMTP relay, in applications like ArcGIS Velocity. Once you specify a name for and generate your API Key, you will finally be provided details about the SendGrid SMTP server address, ports, username, and password (API key). Consider storing this information in a safe and secure location after it is generated. It is important to reference this information when configuring your email output(s) in Velocity. At this point, you can verify your integration.   Store the SMTP address, ports, username, and password Configure an Email output in ArcGIS Velocity   Now that you have established all the necessary SendGrid components, you can access ArcGIS Velocity and begin to configure an Email output that uses the SendGrid SMTP server. Navigate to an existing analytic, or create a new one, to begin configuring an email output. When you add an email output to an analytic, you will need to first open it to configure your SMTP settings.   Add an email output to an analytic from the outputs folder Step one when configuring an email output is to configure SMTP server options. If you refer to the SendGrid settings you stored from earlier, you will want to specify the SMTP Host property as the SendGrid SMTP Server (often smtp.sendgrid.net). By default, you can use port 465 and leave the SMTP server security property set to SSL/TLS. For the Username property, you will need to specify the username provided by SendGrid (often apikey). Lastly, for the Password property, provide your API key that was auto generated earlier.   Configure the SMTP Server with the SendGrid settings At this point, you should be ready to proceed to step 2 where you can configure your email fields. The From email address should be defined as the email you provided as your single sender identity. The remaining properties, To email addresses, Subject, and Message body can all be specified however you wish. You can either specify manual values, as I had done in the case of my Subject and Message body below, or you can reference fields and their values from your data as I had done using my “email_recipient” field for the To email address property below.   Configure email fields with manual entry or data values Once you click Complete, you should be good to go with sending emails as part of your real-time or big data analytic. It is important to remember that the free Email API that SendGrid offers has a cap of 100 emails a day. If you require more than that, it may be worth exploring an alternative subscription plan.   Sending emails is just one of many ways to notify others of critical events. If you are interested in seeing what other outputs are available for sending notifications, or processing data in general, feel free to visit and explore our resource on outputs in ArcGIS Velocity.   ... View more

Have you ever had to employ a file-based output connector such as Write to CSV File to better understand and troubleshoot what is happening with your real-time data as it's processed in a GeoEvent Service? Or rather, have you ever had to publish a pseudo stream service or feature service just to verify the geometry of your event data is as expected? What happens when you forget to configure an output connector while creating a new GeoEvent Service? What do you do when you realize you need to edit an input in a GeoEvent Service before you're ready to publish it? If you're like me, these are just a few hurdles you've encountered when working with different elements in the service designer of ArcGIS GeoEvent Manager. It can take time, patience, and a certain level of familiarity to be successful. We've recognized how many steps you must go through to simply and quickly determine what's happening with your real-time data as it's processed in a GeoEvent Service. Additionally, having to navigate to the multiple pages in GeoEvent Manager to create new or edit existing inputs, outputs, site settings, and more can take time. Given these realities, i'm excited to announce many new usability enhancements as well as a new sampling utility available in the service designer at 10.8. These new capabilities will help you be more efficient and effective when defining your real-time services, making the hurdles mentioned above a thing of the past.   So, let's explore some of these new capabilities in more detail! Working with elements and settings in the service designer First, we’ve made some exciting functionality and usability enhancements to the service designer in GeoEvent Manager. Many of you are probably familiar with the New Element list where you could previously only add new filters and processors to a GeoEvent Service. With 10.8, you now have the ability to create new inputs and outputs as well as copy existing inputs/outputs and configure them directly in the service designer. So if you're in a GeoEvent Service, and you forget to create an input or output connector, there's no longer a need to leave the current GeoEvent Service or open another browser tab just to create it. Simply add an input element from the New Element list to the canvas, configure the properties, and save it. It's the same workflow you've become familiar with when adding and configuring both filters and processors. You may be thinking, “what do I do if I need to edit an input or start or stop that input?” Well, there's now the ability to start, stop, and even edit existing connectors from inside the service designer. Simply right click an input/output you've added to the canvas or one that exists in the element list to access the action menu. For example, in the illustration below, by right-clicking the Receive Flights Data input, you can choose to edit the input's properties, delete the input, fit the input's bounding box to the text, as well as start or stop the input to control the flow of data. By choosing to edit the input properties, a new window will appear allowing you access to all the familiar properties of the input. Should you choose to stop the input, a new status icon on the element will change from green to gray to reflect the stopped state. Quickly edit, delete, start, and stop an input or output by double clicking the element in the canvas. While inputs and outputs connectors are certainly important, they're only half the story when it comes to publishing a working GeoEvent Service. There's also several GeoEvent Server site settings that need to be considered and set appropriately, oftentimes, before successfully publishing a GeoEvent Service. That leads to our next exciting enhancement, the ability to access and edit several key GeoEvent Server site settings directly in the service designer. These settings include access to your GeoEvent Definitions, GeoFences, Data Stores, and Spatiotemporal Big Data Stores.   Have you ever realized halfway through configuring a spatial filter that you forgot to configure your GeoFences? Or, maybe you forgot to create the target GeoEvent Definition for a Field Mapper Processor? Rather than leave your GeoEvent Service, and potentially lose the work you put into configuring it, simply access those settings directly in the service designer. From the Site Settings list, you can double click any of the settings available to open and access those particular settings. When you're finished making any necessary updates, just save your changes and continue configuring your GeoEvent Service. Sample and view real-time data In addition to the above enhancements to the service designer, i'd like to next introduce you to our newest utility in ArcGIS GeoEvent Server 10.8; the GeoEvent Sampler. As the first of its kind, the GeoEvent Sampler is an embedded utility in the GeoEvent Manager's service designer that allows you to quickly sample, review, and even visualize processed data in real-time as it flows through routes and elements in a GeoEvent Service. No longer is it necessary to spend time and effort configuring different types of ephemeral outputs just to review, visualize, or verify your processed data is as expected.   Unlike GeoEvent Logger or GeoEvent Simulator, which are separate Windows applications, GeoEvent Sampler is embedded in GeoEvent Manager. Therefore, you'll be able to use this new utility in both Windows and Linux environments.   Let’s explore how the GeoEvent Sampler can be used to help you build and/or troubleshoot a GeoEvent Service. Verifying your schema Let’s say you want to ensure your Field Mapper Processor is altering the schema of your processed event data correctly. Prior to 10.8, you could write the data emitted from the Field Mapper Processor to an external JSON file. This meant first creating and configuring a separate Write to a JSON File Output Connector. Next, you would need to add that output to your existing GeoEvent Service. Once this was set up, and your outbound data was writing to a JSON file, you would then need to open the JSON file in a text editor to review the data as formatted JSON. While this workflow isn't necessarily difficult to accomplish, it can be time consuming. Using the new GeoEvent Sampler at 10.8, you can simply select the route that connects the Field Mapper Processor to your next element (e.g. an Output Connector as shown in the illustration below) and sample the event records (formatted in JSON) that are being emitted in real-time on that route. A sampled event record allows you to confirm the definition and schema are correct. By never having to leave the service designer, you can quickly confirm whether or not the schema is being updated as expected. If you happen to notice that one of your target definition fields is spelled incorrectly, or that the data type of a field is a string instead of an integer, simply edit the target GeoEvent Definition without leaving the service designer. Remember, GeoEvent Definitions can now be edited directly in the service designer at 10.8 as mentioned above. After making your edits, want to double check that the changes are correct? Just publish the GeoEvent Service and sample the event data on that route again! Another useful scenario where GeoEvent Sampler could come in handy is if you wanted to compare event data on two routes in a GeoEvent Service. For instance, if you wanted to compare your event data's original schema to the schema after it's emitted from a Field Mapper Processor. First, select and sample the first route that's sending the data into the Field Mapper Processor. This will sample the data right before its schema is transformed by the processor. Next, select and sample the route that's emitting data from the Field Mapper Processor for comparison. This will sample the data after the schema as been transformed. With just a few clicks, you can see the data changing in front of you, in real-time. Comparison sampling confirms the source and target fields are mapped correctly. For example, the MPH field (left) was renamed to Speed (right). Verify attribute values GeoEvent Sampler can also be used to verify the attribute values of your real-time data. Continuing with the flight data example in the illustration above, there is a new field called Speed whose value is in miles per hour (e.g. 500 mph). Let's say you want to change the mph value to kilometers per hour (kph). To do this, you need to use a Field Calculator Processor to multiply the mileage value in the Speed field by 1.609344 to get the kilometers per hour (i.e. Speed * 1.609344). While this is easy enough, how can we quickly verify the conversion is happening before configuring the rest of the GeoEvent Service? Prior to 10.8, one option would be to create and configure a Push Text to an External TCP Socket Output Connector and view the output in GeoEvent Logger. Data from before and after the Field Calculator Processor could be routed to the output and viewed in GeoEvent Logger. While this is relatively simple to set up, it does take time, just like the previous JSON file example above. Something else worth considering with this type of workflow is data velocity. Depending on the rate at which your real-time data is being received and processed, it could be a challenge to review the data in GeoEvent Logger since it could be constantly updating with new information. You could close the TCP connection in order to allow you to review the data in GeoEvent Logger, but that's just another factor to consider.  Now at 10.8, you can use GeoEvent Sampler and select the route before the Field Calculator Processor to observe the speed data in miles per hour (mph) and then select the route after the processor to view the speed data after its been converted to kilometers per hour (kph). It's important to note that GeoEvent Sampler is not a logging utility, you can only sample a fixed number of GeoEvents on a route (1, 10, or 100 at a time). So, in the case of the flights example, you could quickly sample a single GeoEvent on each route to verify the processor is correctly calculating the speed in kph. There's no need to comb through hundreds of processed GeoEvents to verify the same thing. The graphic below illustrates this. Comparison sampling of two non-adjacent routes confirm the original MPH field (left) and it's value of 502.0 was correctly converted to kph in the Speed field (right) with a new value of 807.89. Verify geometry So far we've covered how GeoEvent Sampler can be used to review and validate your schema and attributes, but what about the geometry of your processed data? Well, included with GeoEvent Sampler is a capability called the Event Viewer. It can be used to display the geometry of processed event data whose GeoEvent Definition geometry field has been tagged with GEOMETRY. Let's say you're tracking airplanes and using a Buffer Creator Processor to create a buffer for each airplane. You want to ensure the airplanes are being buffered by the correct distance before proceeding with the configuration of the GeoEvent Service. In this case, you need to make sure the point geometry is being changed to polygon geometry. Prior to 10.8, you could verify this a few different ways. First, you could write the buffered GeoEvent to a JSON file and review the geometry object for rings. But even then, how do you know that this geometry will be displayed correctly in a feature or stream service? Unless you're an avid fan of deciphering JSON syntax, you'll likely try to send the data to a feature or stream service to see if it takes. After all, if you can see the geometry of the data display in a web map, you can be confident that buffering is happening, and that the geometry is what you expect. Of course, checking the geometry of a GeoEvent by using a temporary feature or stream service first means creating and configuring those services. Before that can be done, there's GeoEvent Definitions, data store connections, and other factors to consider. Using GeoEvent Sampler at 10.8, you can now sample the GeoEvents emitted from the Buffer Creator Processor. Once you have a sampled GeoEvent, you can then simply open the Event Viewer to display the geometry. If you choose to sample a single route, the Event Viewer will display the geometry of the sampled records from that route. If you choose to sample two routes for comparison, the Event Viewer will display two map views that show the geometry of the sampled data from both routes respectively. In the illustration below, you can see the point geometry of the source flight data and the polygon geometry of the buffered data output from the Buffer Creator Processor. The Event Viewer in GeoEvent Sampler, allows you to confirm the valid point geometry of the source event data and the valid polygon geometry of the buffered event data. These are just a couple ways GeoEvent Sampler can be used to assist with the creation or troubleshooting of your GeoEvent Services. Other example use cases of GeoEvent Sampler not covered here include (but are not limited to) checking datetime values, filtering, regular expressions, construction point geometries, and more. Hope you enjoy using these new enhancements! If you have ideas for future enhancements, please submit those on the Real-Time GIS place on ArcGIS Ideas. ... View more

Back in June of 2019, at the 10.7.1 release of ArcGIS GeoEvent Server, we shared the news of new documentation for the available out-of-the-box input and output connectors. Today, we are happy to announce new and expanded documentation for each of the available processors in GeoEvent Server 10.8. Like the connector documentation, each processor now has expanded information to help you properly configure the processor. Access the new processor documentation by visiting What are Processors? where you can access links to each individual processor you may be interested in. Alternatively, you can access the new processor help directly in ArcGIS GeoEvent Manager when editing or configuring a new processor. Simply click the Help button in the processor property dialog to expand the embedded help. It's important to note that even though this new documentation is available with the 10.8 release, the information is still applicable to previous versions as well. The new processor documentation follows the same format and style as the input and output connector documentation that you may already be familiar with. First, you’ll see a general summary as well as some example use cases for each processor. Beneath that you'll find detailed usage notes. These notes are intended to provide some extra contextual information for successful configuration of each processor. Following the usage notes, is a parameters table that includes detailed information about the parameters for each processor, what each parameter does, options to configure it, and more. The parameter table list all of the available parameters, meaning you can learn about all the parameters that are shown by default in additional to all of the conditional parameters that are initially hidden due to their dependencies on other parameters. We recognize that many of the available processors have their own nuances and quirks, so we’ve included a final section that details various considerations and limitations. This information is intended to provide some additional context about how the processor fundamentally works, and therefore, will hopefully help guide how you approach incorporating it into your GeoEvent Services. As with our input and output connectors, you can find step-by-step tutorials on how to configure many of the available processors on the ArcGIS GeoEvent Server Gallery. ... View more