Latest Contributions by EricIronside

PreCise MRM, LLC - A Subsidiary of FORCE America, Inc. 1311 East Franklin Road, Suite 101/102 Meridian, ID 83642 www.precisemrm.com 1-888-449-0357 ... View more

There are several situations where you would want to use the Watch a Folder for New [Format] Files input to read data files from your file system into GeoEvent.  Typically, I see this input used when GeoEvent doesn't supply an Input Connector for the specified situation (e.g. an FTP transport).  To get around this, people will write a script that will gather the external data, write it to a file, then expect GeoEvent to import that data. However, some report issues in this approach as files will get skipped, or data corrupted when it is read into GeoEvent.  This blog presents a way around that issue: Setting up GeoEvent Set your GeoEvent Watch a Folder for New [] Files input up to read files with an extension of .csv (or any extension that meets your needs, so long as it is different from the extension used below). CSV Files - Advanced Input Properties Depending on the data contained in your CSV file, you might need to change some of the Advanced properites on the input: If you deploy the input as is without changing the defaults, it will assume the GeoEvent Definition name is the first item in the CSV record (line). So if that name doesn’t match a definition in the Site > GeoEvent > GeoEvent Definitions list you will get errors like the following: Starting to read file "TestDataLookup.csv"... Failed to translate an event. The GeoEvent Definition "Radio_Name" was not found. Failed to translate an event. The GeoEvent Definition "AK487" was not found. Finished reading all of the lines "2" in the file "TestDataLookup.csv" in 3 ms. Under the Advanced properties for the input, you can change the ‘Incoming Data Contains GeoEvent Definition’ option to “No” IF the user has NOT already created the GeoEvent Definition for the csv file, change ‘Create Unrecognized Event Definitions’ to Yes IF the user HAS already created the GeoEvent Definition for the csv file, leave the  ‘Create Unrecognized Event Definitions’ at No and select the definition name they have already created. Writing Data Files to Disk When you write the data (from your script, application, or other means): Write the data to a file on disk with a different extension than your GeoEvent input is expecting, such as .txt  Flush/close the file to ensure the data has been fully written. Change the name of the data file to match the extension GeoEvent is expecting. In this example, change the extension .txt to .csv Since changing a file name on the OS is atomic, this process guarantees that GeoEvent won't try to start reading the data until it has been fully committed to disk. Please note that #3 only works if you use a RENAME or MOVE (updating the file path/name). You cannot use a COPY  for #3 because this just ends up copying the bytes again and you will run into the same original problem. Attached to this post is a sample script that shows how to do this in python. ... View more

GeoEvent Server 'Global Settings' provide a way in GeoEvent Server to review and modify global configuration settings. These property settings apply to the entire GeoEvent Server configuration, and may be used as defaults in various places. For example, when a GeoEvent Connector needs to define SMTP settings for sending an email, it will start with the global system setting values, then allow you to modify those settings for the specific component you are creating. SMS Settings Carriers You can add additional carriers by entering a Key / Value pair and pressing the 'Add' button. Key is case-sensitive and can contain upper/lower case letters, numbers, some special characters, and spaces. Keys should be unique, so be sure you enter a different key for each character. The value is a bridge address (see below) that should start with the '@' character and should conform to email address specifications. Each carrier has a bridge address (starting with the '@' symbol) that the message is sent to. This address is appended to each recipient's phone number and the SMTP server sends an email to that carrier's email bridge. The carrier's bridge then delivers the message to the recipient as a text message. By default the following carriers are provided AT&T (@txt.att.net) Altel (@message.alltel.com) Boost Mobile (@myboostmobile.com) Metro PCS (@mymetropcs.com) Nextel (@messaging.nextel.com) Powertel (@ptel.com) Qwest (@qwestmp.com) Sprint (@messaging.sprintpcs.com) Sprint PCS (@messaging.sprintpcs.com) Suncom (@tms.suncom.com) T-Mobile (@tmomail.net) US Cellular (@email.uscc.net) Verizon (@vtext.com) Virgin Mobile USA (@vmobl.com) More information on Gobal Properties may be found here:  http://enterprise.arcgis.com/en/geoevent/latest/administer/managing-global-settings.htm ... View more

GeoEvent Server 'Global Settings' provide a way in GeoEvent Server to review and modify global configuration settings. These property settings apply to the entire GeoEvent Server configuration, and may be used as defaults in various places. For example, when a GeoEvent Connector needs to define SMTP settings for sending an email, it will start with the global system setting values, then allow you to modify those settings for the specific component you are creating. SMTP System Settings The following system settings may be helpful in pre-configuring your SMTP server settings or adding a new text messaging carrier.  If you intend to create several text or email outputs, you should modify these settings before you create your outputs to streamline your configuration. SMTP Settings Default SMTP server host The SMTP server to use to send the text messages. This will typically be your corporate email server (e.g. smtp.companyx.com). If you have problems, or are using an external SMTP email server (like GMail), please see RJ Sunderman's reply to the following post: I Am Not Receiving Any Emails Default SMTP port The port the SMTP server listens on. The default value for SMTP servers are: Using anonymous authentication is 25. For SSL/TLS, the default should usually be 465. Other common SMTP ports are 2525 and 587. You will need to contact your IT administrator to identify the port(s) over which SMTP messages can be sent in your organization. Note also that external ports may be blocked by your firewall, so check with your IT group. Default SMTP authentication Anonymous The email server will send anything so long as the Sender's email is valid on that server. If the Sender's email address is not valid the email will be discarded without error. SSL (Secure Socket Layer) - an earlier version of encryption. TLS (Transport Layer Security) - a later version of encryption that replaced SSL. Default SMTP user name [optional, used only with SSL/TLS authentication modes] Enter a username that can be used to authenticate with the SMTP server. Default SMTP password [optional, used only with SSL/TLS authentication modes] Enter the password for the username entered above. More documentation on GeoEvent Server Global Settings may be found here: http://enterprise.arcgis.com/en/geoevent/latest/administer/managing-global-settings.htm ... View more

As RJ Sunderman mentioned in his blog, automating administration of the GeoEvent Server using python scripts can be a powerful way to augment the way you manage your real-time system (such as stopping outputs before a reboot or starting those outputs after a restart). a blog post by Andy Ommen has an excellent description of how to do this (I've even updated his script here) - Scripting tasks using the GeoEvent Admin API Scripting can also be useful for those times when GeoEvent needs a bit of help. For example if a property on an input or output must be periodically updated. One of those situations recently arose when a user's AVL provider required a URL Parameter to supply a timestamp for the data being requested.  It is this scenario that I'll use as an example of how to modify properties of a GeoEvent component using a script.  This methodology could be used to update any of the editable properties on any of your GeoEvent components that allow modification, as Jake Skinner shows in his article on updating a token. An Input's "since" property... An AVL provider supplies a HTTP RESTful API for polling vehicle data in JSON format.  Yeah! The good news is that GeoEvent provides an Input Connector that does exactly that.  But on further reading you find that the API expects the polling request to include a 'since' parameter in the URL. An example of the expected format is below: https://avl.trackingcompany.com/api1/json/currentLocations?user=john&pw=secret&since=2012-10-08T09:20:30Z  Unfortunately, GeoEvent has no way to automatically update that date next to the 'since' parameter.  So without any other solution, you'd be stuck manually updating the GeoEvent Input's properties every day (or every hour). What we need is a way to set that timestamp to something recent, so we can reduce the amount of duplicate data we recieve and reduce the load on both servers. The GeoEvent Admin API to the rescue... Fortunately, the GeoEvent Admin API allows us to update an input programatically and we can automate this using a little python scripting.  To see what is offered from the GeoEvent Admin API, you can view the Swagger documentation on your GeoEvent Server by going to the following URL. https://<YourGES>:6143/geoevent/admin/api/index.html  Using this API we can do operations like get, add, update, delete, start, and stop on most of the compents within GeoEvent.  So conceptually, to solve the problem above, we just need to: get the input with the 'since' parameter, set the value of that parameter, and update the input.  Step by Step Create your GeoEvent Input In GeoEvent create a new 'Poll an external website for JSON' Input.  Be sure to split the URL path from the parameters by using the Input's 'URL' and 'Parameters' properties.  URL Paramters are any part of the URL after the '?'. For example    Given the URL:             https://myserver.domain.com/api1/json/gateway?user=john&since=2019-03-21T14:22:12Z&pw=secret    Your Input properties should look like this:         Identify your GeoEvent Input By default the script will look through all inputs for URL Parameter properties containing a 'since' parameter.  You can help the script narrow its search by providing the 'Name' of the input(s) you want it to look at.  To do this, go into your GeoEvent's REST end poing and select your input.          https://<yourGES>:6143/geoevent/rest/  After you click on your input, you will see the list of properties.  Note and/or Copy the value for the 'Name' property. So in this case, we want to copy and save the 'Name' value "a0e151b8-0c00-4c12-838d-16cdefb36ba8". Set up the script You can get the script from this GitHub repository. You want to download the file 'UpdateGEEInputURLParam.py'. Once downloaded, the following parameters will need to be set before the script will run properly. server_fqdn – This is the fully qualified domain name of the server where GeoEvent is installed. Example: server.domain.com username – This is the username for the GeoEvent Server Manager login credentials password – This is the password for the GeoEvent Server Manager login credentials geeTargetInputList – This is a list of ‘name’ strings (leave empty if you want the script to look at all inputs). This is where you'd paste (inside of quotes) the 'Name' parameter of your input above. Other script parameters you may update urlParamName – This is the property on the URL Parameters you want to change. The default is ‘since’ to match our context, but you might change this to anything that matches your use case. Examples: timeSince, token, startTime, ... time_offset_sec – This is the amount of time to add (or subtract if you supply a negative number) to the current time. Positive numbers will add time the current time. Example: now() + 30 seconds.  Negatvie numbers will subtract time from the current time. Example: now() - 60 seconds. The default is -60 seconds (subtract 1 minute from the current time stamp) Script constants There are a number of constants that follow the parameters above that shouldn't be modified in most cases.  However, if you intend to extend this script, or have a non-default installation of GeoEvent, you might need to add or modify some of these. Running the script Installing The script requires an external library to be installed into your Python before it will run correctly.  The library is located here in GitHub, but to install, all you need to do is the folloiwing: Open a command prompt on your machine Navigate (cd) to the scripts directory in your Python2.7 install (usually this is C:\Python27\ArcGISx6410.6\Scripts) Type ‘pip install arcrest_package’ and hit enter Testing If you are editing the script in an IDE you will be able to run the script there.  Outside of the IDE, you will want to be sure that Python is installed on your machine and is executable (on Windows, you can register .py files to run as python.exe files, or you can add the folder containing the python.exe to your system's PATH variable). This script can be run/tested on any machine that has network access to the GeoEvent machine. As the script runs, it will print out log messages indicating the input(s) it is modifying:          Looking at clientParameters on input named 'a0e151b8-0c00-4c12-838d-16cdefb36ba8'          Updating clientParameters 'since' parameter value =2019-03-21T04:43:46Z to 2019-03-21T14:22:12Z       Updated clientParameters 'since' parameter on input named a0e151b8-0c00-4c12-838d-16cdefb36ba8 Checking your Inputs, the Parameter property should be updated and the value of the 'since' parameter will be the current time minus 1 minute. Scheduling You can use a scheduler (such as Windows Task Scheduler) to run your script at a specified time or interval (e.g. hourly).   ... View more

This blog post shows how to set up PKI authentication in a GeoEvent Data Store connection.    Install Certificates The first thing you will need to do is install certificates for both your ArcGIS Enterprise machine and your authentication provider.  Export Certificates The easiest way to get these certificates is to export them from your browser.  On your GeoEvent machine, open a tab in your browser and enter the URL for each into the address bar.  Once you’ve done this, you can access the certificate by clicking on the ‘lock’ symbol in the address bar and viewing the certificate.   Once you are viewing the certificate, go to the ‘Details’ tab and press the ‘Copy to File…’ button. This brings up the Certificate Export Wizard Press ‘Next’ to go to the export format page, accept the default. Press ‘Next’ to go to the file to export page. Press the ‘Browse’ button and select a location and file name for the exported certificate. Press the ‘Next’ button and then press the ‘Finish’ button.  The wizard should notify you that the file was exported.  Complete this process for both your ArcGIS Enterprise and your security provider URL. Import Certificates On your GeoEvent machine, open the ArcGIS Server Admin console by going to https://<geoeventserver>:6443/arcgis/admin. Click on ‘machines’ then click on the link for your machine. At the bottom of the next page, click on the ‘sslcertificates’ link. On the SSL Certificates page, press the ‘importRootOrIntermediate’ link to open the import page. Enter an alias for your certificate this could be the fully qualified domain name of the server the certificate represents avoid special characters and spaces, periods are ok.  Press the ‘Browse’ button and locate the .cer file you saved above. Complete this import process for both certificates. When you are done, your list of SSL Certificates should include a link for both your ArcGIS Enterprise and the security provider servers. Restart GeoEvent Finally, you will need to restart your Windows services. To do this either reboot your machine, or: In Windows Services Stop ArcGIS GeoEvent Restart ArcGIS Server, wait about 30 seconds Start ArcGIS GeoEvent, wait about 60 seconds   Create PKI GeoEvent Data Store Connection Store & Register Your PKI Certificate(s) On your GeoEvent machine, create a folder where you will store the PKI certificates.  These certificates represent the user you will be connecting to your enterprise as.  The path MUST be on the local GeoEvent machine, UNC paths are not allowed. This folder can contain one or more user certificates (.pfx files), where each certificate represents a different user.  Log in to GeoEvent Manager and go to the ‘Site | GeoEvent | Data Stores’ page.  Press the ‘Register Folder’ button. On the Register Folder dialog enter the path to the folder you created above to store your certificates.   Configure GeoEvent Data Store Connection Within GeoEvent Manager, on the same Data Stores page, press the ‘Register ArcGIS Server’ button to open the ‘Register ArcGIS Server’ dialog.  On the dialog, select the ‘Portal’ option at the top.  Next, select the ‘Use PKI’ option to check that box.   Name - Enter a name for the GeoEvent Data Store connection (you can use spaces, I typically also include the name of the user the connection will use).  URL – The URL of the ArcGIS Enterprise system you are connecting to.  As shown in the example, this should be the Enterprise Server’s Web Adapter URL (without the port number) and not the direct administrative connection (with the port number). PKI Data Store Folder – Select the folder you created above to store your PKI certificates. PKI Filename – Enter the name of the certificate file you wish to use. This is case sensitive, and you should include the file extension (example: User1.pfx) PKI Password – Enter the password that is protecting the .pfx file. Note that this is not the same as the user’s password.  It is the password used to access the .pfx file contents (this was set when the .pfx file was created). Check with your certificate manager who created the .pfx file if you don’t know this password. Discovery Rate – The default is fine. This is the rate that GeoEvent will reach out to the ArcGIS Portal to refresh GeoEvent’s catalog of the user’s content on that enterprise site.   Notes on GeoEvent Data Store connections Each connection represents a specific user. Thus, the content that is accessible by a GeoEvent Data Store connection is limited to: Public items Items owned by that user.  Items shared with the user (e.g. through groups) will not be visible/accessible.  Because of this, it may be required to create multiple GeoEvent Data Store connections, each representing a different user.  For this reason, I usually include the user’s name in the GeoEvent Data Store connection’s name.  I usually suggest that organizations dedicate one “Application” user to GeoEvent and publish all content that GeoEvent will need access to under that one headless user.  However, this is not always possible. ... View more

GeoEvent provides a wide array of outputs, but the Text and Email Message outputs are somewhat unique in that they are sending data directly to a human.  Because of this, it is important to be able to structure these output messages in a manner that is precise and actionable.  This blog post will go through the aspects of creating a meaningful and concise text message output. Topics in this post: The 'Send a Text Message' output is a “fire-and-forget” type of operation. The GeoEvent Server will not contact the specified SMTP server to determine its status, and there is no way for GeoEvent to process acknowledgement or error messages which might be sent from the SMTP server. You cannot assume that because an output incremented its event count that a text message has been successfully delivered or received. The incremented event count only indicates that a message has been succesfully sent to the specified carrier's server for delivery to the specified address (mobile phone number). Creating a Text Message Output Start by going to the 'Services | Outputs' page in GeoEvent Manager and pressing the 'Add Output' button.  This will bring up the Output Connectors dialog.  In the search bar, type "text m" and press the 'Select' button on the "Send a Text Message" item. This brings up the "Creating Output - Send a Text Message"  Standard prameters  Name [Required] The name should describe the output text message. Names should be unique (but don't have to be). You can use spaces and other special characters if you wish. I typically use the original default name and append (or prepend) a descriptive name. I also like to group my input, service, and output names by using group/project/topic names first, then more descriptive names after that. Some examples:  avl-sms-text-out-notification-driver-idle avl-sms-text-out-warning-driver-speeding safety-sms-text-out-urgent-driver-trackgap safety-sms-text-out-warning-driver-outofzone Sender's email address [Required] One email address that will be used to send the text message. This email must be a valid email address on the SMTP server (see advanced parameters described below). So long as the email address is valid for that server, you can use any email you wish. You can use event fields to dynamically set the sender's email Enclose the field name containing the sender's email address like this ${FIELD_NAME}. For example, if each event contained a field named "manager_email" then you could dynamically set the value of the sender's email address by setting this property to Message [Required] The message body should be plain text text messages can't use rich text or html tags. Anything that looks like a URL might be interpreted as such (this depends on your text messaging app). For me: This works: "https://www.esri.com/en-us/home" This works: "www.esri.com" This does not: "<a href='https://www.esri.com/en-us/home'>Esri</a>" Currently there is no way to embed local application calls in a text message (e.g. "arcgis-survey123://?itemID=123456789abcd1234567890abcd") To do this, use the 'Send an Email' output You can use event fields to dynamically set the message Enclose the field name(s) containing the recipient's phone number(s) like this ${FIELD_NAME}. For example, if each event contained fields named "fleet_id", "vehicle_vin", "vehicle_speed", "gps_datetime" then you could dynamically set the value of the message by setting this property to Carrier [Required] Choose a carrier from the list of providers. If your carrier is not listed, you can modify this list in the setting section (see the System Settings section below). Please note that each 'Send a Text Message' output can only support a single carrier. If your organization supports multiple carriers, you will need to set up one output for each carrier. After you configure the first text message output, use the 'Copy' button on the 'Outputs' screen to duplicate your output for the other carriers.  Recipients [Required] This is a comma separated list of cell phone numbers to send the text message to. You must provide at least one (1) recipient. Using characters other than numerical digits or commas will result in a 'Error occured while sending SMS: No recipient addresses' error. The output is not able to validate the recipient's phone number with the carrier.  GeoEvent will not be able to determine if a phone number is incorrect, disconnected, or not supported by that carrier (see common errors below). You can use event fields to dynamically set the recipient Enclose the field name containing the recipient's phone number(s) like this ${FIELD_NAME}. For example, if each event contained a field named "cell_number" then you could dynamically set the value of the recipients by setting this property to Advanced Properties Note, these properties can be set ahead of time by updating the GeoEvent Settings (see the System Settings section below). SMTP server host name [Required] The SMTP server to use to send the text messages (Simple Mail Transport Protocol = Email Server) This will typically be your corporate email server (e.g. smtp.companyx.com). If you have problems, or are using an external SMTP email server (like GMail), please see RJ Sunderman's reply to the following post: I Am Not Receiving Any Emails SMTP server port number [Required] The port the SMTP server listens on. The default value for SMTP servers are: Using anonymous authentication is 25. For SSL/TLS, the default should usually be 465. Other common SMTP ports are 2525 and 587. You will need to contact your IT administrator to identify the port(s) over which SMTP messages can be sent in your organization. Note also that external ports may be blocked by your firewall, so check with your IT group. SMTP authentication mode [Required] Anonymous The email server will send anything so long as the Sender's email is valid on that server. If the Sender's email address is not valid the email will be discarded without error. SSL (Secure Socket Layer) - an earlier version of encryption. TLS (Transport Layer Security) - a later version of encryption that replaced SSL. User name [optional, used only with SSL/TLS authentication modes] Enter a username that can be used to authenticate with the SMTP server. Password [optional, used only with SSL/TLS authentication modes] Enter the password for the username entered above. Global Settings The following GeoEvent Server global settings may be helpful in pre-configuring your SMTP server settings or adding a new text messaging carrier.  If you intend to create several text or email outputs, you should modify these settings before you create your outputs to streamline your configuration. GeoEvent Server Global Settings: SMTP [Email Server] Settings GeoEvent Server Global Settings: SMS [Text Messaging] Providers Common Errors "Error occured while sending SMS: Unknown SMTP host: smtp.HOST-NAME.com" Check the host name you are using because the one you provided cannot be found, is not responding, or is not correct. "Error occured while sending SMS: Could not connect to SMTP host: smtp.HOST-NAME.com, port: ##" Check the port that your server is monitoring, it may be wrong Port 25 is typical for anonymous authentication Port 465 is typical for SSL/TLS Please note that the host name you are using was found (because it tried the port on that server). Sender gets an email response from the carrier that destination is invalid: One or more recipients is invalid or incorrect. Check the phone numbers of the recipients and/or the format of this property value. GeoEvent says it sent the message, there are no errors in the GeoEvent Server logs, but the text message was not recieved. Check the Sender's email address and verify it is a valid email address on the SMTP server. Check the phone numbers of the recipients to verify they are correct. Romove any characters from the recipients property that are not digits or a comma (spaces, dashes, etc.) Your firewall may block outbound communication on the configured port (talk to your IT about what ports are allowed/blocked). Your external SMTP provider may require additional settings to allow you to use it from GeoEvent.  For example, please see Hugo Gaignard's reply to the post I Am Not Receiving Any Emails for Google security settings. ... View more

For the ‘Receive New Data Only’ property to work on the 'Poll an external website for ...' inputs, the external site has to implement that functionality via the response header 'Last-Modified' (see '14.29 Last-Modified'). If the external site doesn’t support/implement this response header then checking the ‘Receive New Data Only’ parameter 'Yes' won’t have any affect and all data that is returned by the external server will be processed. Please also note, that setting this propert to 'Yes' does not cause the GeoEvent Input to set any request parameter/header to indicate to the external server that only new data should be returned. So it is up to the external website at the provided URL to implement the time window that new data is returned for (in addition to setting the 'Last-Modified' header value). ‘Receive New Data Only’ = No Every response from the external server will be accepted and all data within each response will be processed. ‘Receive New Data Only’ = Yes & 'Last-Modified' IS Supported IF 'Receive new Data Only' is 'Yes' AND the external server provides a response with a datetime value for the header name 'Last-Modified' (e.g. "Last-Modified: Tue, 15 Nov 1994 12:45:26 GMT") THEN that lastModifiedDatetime will be used to determine if the response is processed or not: If lastModifiedDatetime is after the cachedLastModifiedDatetime the response data will be accepted and the input will set the cachedLastModifiedDatetime = lastModifiedDatetime. If lastModifiedDatetime is before the cachedLastModifiedDatetime the response data will be ignored. Please note: it is up to the external server to respond with a subset of records that have been modifed since the lastModifiedDatetime. This might be done via the specified URL (e.g. http://myserver/getNewValues) or a URL parameter (e.g. '?NewValuesOnly=true'). The time window to filter modified results is entirely up to the external server. If the geoEvent polls every 30 seconds, but the server only returns modified results for the last 5 minutes, then the input may have to process 5 minutes worth of data every 30 seconds (assuming the data is changing more frequently than 30 seconds). ‘Receive New Data Only’ = Yes & 'Last-Modified' is NOT Supported This situation will behave the same as setting the ‘Receive New Data Only’ = No. Unfortunately, the workarounds for this situation all depend on the external site: Modify the external server to support the 'Last-Modified' response header parameter. Implement on the server URL site a request URL parameter of ‘newvalues=true/false’ If set to true, then the external server will return updated records only within an arbitrary time window. This is a bit messy because GeoEvent can’t control the polling/window of the last modified date. Another [potentially better] option would be to implement on the server URL site a request URL parameter for a defined window of updated values ‘newvalues=#’, where the numeric value references a defined time value (e.g. minutes). If omitted or <=0 all values are returned If >0 then any value that has been modified within that number of time units is returned This allows GeoEvent to control the polling/window of the last modified date. Use a filter in your GeoEvent Service to filter out older records This assumes the data returned by the server contains a date field that can be used as a "Last Modified" date (if not, you are out of luck) Add this filter immediately after your input to avoid processing unchanged records. Finally As a final note, there is no way to dynamically change the 'Parameters' property of the GeoEvent Input.  So trying to set a dynamic/calculated datetime on the requet URL parameter will not work (e.g. 'Parameters' = "lastModifiedDate > ( now()+30)" ). ... View more

The input ‘Receive JSON on a REST Endpoint’ can be used to allow external data providers to push events into GoeEvent. As the name implies, the input presents an HTTP transport that implements a REST interface which allows data to be POSTed.  The adapter for this input is the JSON adapter (XML, GeoJSON, and Esri Feature are other adapter options). Any valid JSON structure can be converted by the adapter into events.  This post reviews the various settings and options when setting up this input.  By default, this input expects a POST method will be used to post raw data.  The GET method may be used along with URL parameters (see the Advanced properites settigns below).   Setting Properties  Shown Properties The following properties should be configured. Name The name of this input (underscores and dashes are ok; you can use spaces, but I avoid them) I typically add to the beginning or end of the default name (e.g. rest-json-in-gps-location or gps-location-rest-json-in) JSON Object Name If blank, the top-level incoming json object (or array of objects) will be converted to a geoevent. If not blank, the incoming json data will be searched for fields with this name, and the contents of those fields will be converted to GeoEvents. Create GeoEvent Definition: True When set to True, a new GeoEvent Definition will be created using the first incoming JSON object as a model See learning mode if the JSON objects vary. When set to False, You must select a GeoEvent Definition to put the incoming JSON objects into. Note that GeoEvent Definitions are case sensitive. GeoEvent Definition Name (New): Generated-GeoEventDefinition A string name for the new GeoEvent Definition If the definition is already created by this input, then that definition is used. GeoEvent Definition Name (Existing) Select from the current list of GeoEvent Definitions known to the system. Data will be marshalled into this GeoEvent definition GeoEvent Definitions are case sensitive.   Advanced Properties The following properties are under the Advanced tab. Default Spatial Reference Specify the default WKID or WKT for geometries that do not contain spatial reference. Or the field that contains the WKID or WKT in each event. Acceptable MIME Types (Server Mode): application/json Comma-separated list of MIME Types that are acceptable to the REST Transport in Server Mode. This input works with JSON so the value is defaulted to application/json Expected Date Format If left empty, the ISO8601 date format is used. Otherwise, the date format is specified in a format string using standard Java conventions: y - Year. M - Month in year. d - Day in month. H - Hour in day (0-23). h - Hour in am/pm (1-12). m - Minute in hour. s - Second in minute. S - Millisecond. z - Time zone (General time zone). Z - Time zone (RFC 822 time zone). X - Time zone (ISO 8601 time zone). a - am/pm marker. G - Era designator (AD). w - Week in year. W - Week in month. D - Day in year. F - Day of week in month. E - Day in week. k - Hour in day (1-24). K - Hour in am/pm (0-11). Y - Week year (2009, 09). u - Day number of week (1=Monday,...,7-Sunday). An example of a common date format would be: MM/dd/yyyy HH:mm:ss.SSS Z Construct Geometry From Fields: False If set to False, the incoming events will either already have a geometry defined or will not have a geometry. If set to True, the values in the provided fields will be used to create a point geometry. The X and Y Geometry Field parameters are required. X Geometry Field The name of the field that has the X (or Horizontal, or longitude) part of the Geometry This property is required if Construct Geometry is True. This property is case sensitive. Y Geometry Field The name of the field that has the Y (or Vertical, or latitude) part of the Geometry This property is required if Construct Geometry is True. This property is case sensitive. Z Geometry Field The name of the field that has the Z (or Depth or Altitude part of the Geometry). This is optional. If blank the z value will be set to 0 Learning Mode: False If False, the schema of the first event will be used to build the GeoEvent Definition. It is assumed the schema of all subsequent events is the same. Additional field values not in the definition will be ignored. If True, the GeoEvent Definition will be enhanced with each new message received. Please notice that the learning mode will alter the GeoEvent Definition so that some of the downstream processes may encounter issues when the GeoEvent Definition is altered. Also, the learning mode should not be used in production as it involves extra overhead. If learning mode is used, you should briefly turn it on until a representative sample set of data has been received, then turn it off. Get Request Contains Raw Data: False This property applies to the Server Mode (Receive Mode) with a client’s Get Request (instead of a post). If False, and a GET method is used to push data, the name-value pairs of request parameters will be converted to JSON. If True, and a GET method is used to push data, The raw data, is assumed to be the value of a single query parameter, The data can be either a JSON String or a CSV string. Only one query parameter will be considered as the raw data. This input assumes the parameter data is JSON. Parameter Name for the Raw Data If the user selects True for Get Request Contains Raw Data then this property defines the name of the URL parameter that contains the JSON data. Using the Input Once you set the properties of the input and press the 'Save' button, your REST endpoint will be created. If you re-open the new input you will see the URL property is populated for you. Please note that either the HTTP or HTTPS format of this URL will work. For example, both of the following should work:                http://<yourserver>:6180/geoevent/rest/receiver/<inputname>                https://<yourserver>:6143/geoevent/rest/receiver/<inputname>  Using your favorite test application, you can use this URL to POST JSON data to your new input.   Use one of the URLs above and set the method to POST. Set the header value ‘Content-type’ to ‘application/json’. Enter some Json into the Raw Body Press ‘Send’ You should see the counter in GeoEvent for the input increase by 1 on the GeoEvent Manager Monitor page. If you set the input to create a new GeoEvent, go to GeoEvent Definitions to check and see if it was created correctly. Other Properties (don't worry about these)  Hidden Properties The following properties are hidden or read only and cannot be changed. Mode: Server This setting tells the HTTP transport that it is hosting the end point and should accept REST connections when data is POSTed (pushed). URL: http://yourserver:6180/geoevent/rest/receiver/inputname This property will store the URL of the input that is used to push data to. This property is read only and will only show up after the input is created The default values are:http://<yourserver>:6180/geoevent/rest/receiver/<inputname>                https://<yourserver>:6143/geoevent/rest/receiver/<inputname>      Unused/Hidden Properties The following properties for the transport/adapter are hidden and not used/applicable. Use Long Polling: False Frequency: 5 seconds Receive New Data Only: True Post body MIME Type HTTP Method: Get Header Parameter (name:value list) Post From: Content Body Content Body Post Parameters Parameters URL Proxy Use URL Proxy: False Acceptable MIME Types (Client Mode) HTTP Timeout (in seconds): 30 Append to the End of Payload As GeoJson: False ... View more

Sharepoint calendars offer an RSS feed that can be consumed by GeoEvent.  This post will show how to parse this specific input.  The input connector 'Receive RSS' will handle subscribing to an RSS feed and parsing out the data into individual events.  In the RSS specification, each <item> segment will be converted to a GeoEvent event.  The exact structure of your RSS feed (in XML) may differ a bit (all sub-segments of item are optional), but in general it will contain the following XML segments: Element Description title The title of the item. link The URL of the item. description The item synopsis. author Email address of the author of the item.  category Includes the item in one or more categories.  comments URL of a page for comments relating to the item.  enclosure Describes a media object that is attached to the item. It has 3 sub-segments: url - where the enclosure is located length - how big the enclosure is in bytes type - a MIME type for the enclosure guid A string that uniquely identifies the item.  pubDate Indicates when the item was published.  source The RSS channel that the item came from.  For the Sharepoint Calendar RSS feed, an example of the raw data is provided below. Notice that the <descripion> segment contains more than just a description. We'll have to parse that data out. <?xml version="1.0" encoding="UTF-8"?> <!--RSS generated by Microsoft SharePoint Foundation RSS Generator on 2/19/2019 9:08:00 AM --> <?xml-stylesheet type="text/xsl" href="https://community.esri.com/services/ServiceCenter/_layouts/15/RssXslt.aspx?List=12345678-abcd-ef56-123a-b3c5d6e7f8ab" version="1.0"?> <rss version="2.0"> <channel> <title>Service Center: Calendar</title> <link>http://serverfqdn/services/ServiceCenter/Lists/Calendar/calendar.aspx</link> <description>RSS feed for the Calendar list.</description> <lastBuildDate>Tue, 19 Feb 2019 17:08:00 GMT</lastBuildDate> <generator>Microsoft SharePoint Foundation RSS Generator</generator> <ttl>60</ttl> <language>en-US</language> <image> <title>Service Center: Calendar</title> <url>http://serverfqdn/services/ServiceCenter/_layouts/15/images/siteIcon.png</url> <link>http://serverfqdn/services/ServiceCenter/Lists/Calendar/calendar.aspx</link> </image> <item> <title>Test Title</title> <link>http://serverfqdn/services/ServiceCenter/Lists/Calendar/DispForm.aspx?ID=60</link> <description><![CDATA[<div><b>Location:</b> San Francisco</div> <div><b>Start Time:</b> 2/19/2019 11:00 AM</div> <div><b>End Time:</b> 2/19/2019 12:00 PM</div> <div><b>Description:</b> This is a test description. There are a few line breaks here. <b>Hello World</b></div> <div><b>Category:</b> Meeting</div> ]]> </description> <author>Person Author</author> <enclosure url="/services/ServiceCenter/_vti_bin/owssvr.dll?CS=96001&amp;Cmd=Display&amp;CacheControl=1&amp;List={12345678-abcd-ef56-123a-b3c5d6e7f8ab}&amp;ID=60&amp;Using=%2Fservices%2FServiceCenter%2FLists%2FCalendar/event.ics" type="text/calendar" /> <category>Meeting</category> <pubDate>Tue, 19 Feb 2019 17:03:46 GMT</pubDate> <guid isPermaLink="true">http://serverfqdn/services/ServiceCenter/Lists/Calendar/DispForm.aspx?ID=60</guid> </item> </channel> </rss> Here's what I did to read/parse this feed: Create a 'Read RSS' input and let it read some events to create the GeoEvent Definition for your feed. Note, you should also be able to use a 'Poll an external website for XML' input, that is what I tested with In my case, the GeoEvent Definition looked like the following (Note you may be missing the author and enclosure segments, and the guid may be a string instead of a group). I created a copy of this definition and flattened it to look like the following Next create a GeoEvent Service Add the RSS input Add an output (can be anything, I used TCP out and wrote to the logger) After the input add a 'Field Mapper' processor to map from your input definition to the flattened definition Next, remove the HTML tags from the description using a Field Calculator processor  Expression = replaceAll(desc, '<[^>]*>', '') Calculate the location Expression = trim( substring(desc, 9+indexOf(desc, 'Location:', 0), indexOf(desc, 'Start Time:', 0) ) ) Calculate the Start Time Expression = trim( substring(desc, 11+indexOf(desc, 'Start Time:', 0), indexOf(desc, 'End Time:', 0) ) ) Calculate the End Time Expression = trim( substring(desc, 9+indexOf(desc, 'End Time:', 0), indexOf(desc, 'Description:', 0) ) ) Caculate the Description Expression = trim( substring(desc, 12+indexOf(desc, 'Description:', 0), indexOf(desc, 'Category:', 0) ) ) Calculate the Category Expression = trim( substring(desc, 9+indexOf(desc, 'Category:', 0), length(desc) ) ) The final GeoEvent Service looks like this: At this point, You can remove line breaks by adding the following around the existing calculations above: replaceAll( [exisitngcalc], '\n', '') You could use the 'Reverse GeoCode' processor to look up the location or a field calculator if location contains an x,y point. Note, that your TRACK_ID may be the author instead of the link because each calendar item will have a unique link, but may be authored by the same person. Finally, the start and end datetimes are strings. Currently (10.6.1) there is no way to parse these strings into dates (this may be a feature in upcoming releases). ... View more

Hello. Is there a way to pass, via the URL, a specific record ID for the Data Page to select?  For example:  I have a survey record that was collected with GlobalID '12345'. It would be nice to be able to provide a link to the data page that would automatically pre-select that survey. Something like:  https://survey123.arcgis.com/surveys/54321abcdefghi/data?id=12345 ... View more

I have an operational layer that i have configured the info window popup to show/hide fields and include a URL for that specific item on the operational layer.  However in the Workforce workers mobile app, the attribute list always displays all attributes for a feature and does not show the customized URL link. Thanks in advance ... View more

Is there a way to tell Survey123 NOT to include a field in the feature service schema? I have several fields used as intermediate calculated values to display a URL for a user to click on. I want to ignore/omit all of the intermediate fields when submitting the data (I don't need to store the URL nor the intermediate data and it is causing errors when pulling the data using another client). Thanks! Eric ... View more