Need documentation / samples for Field Calculator and Regular Expression notation

[BruceDodson]

Having reviewed the 10.2 final help, and the most recent beta training materials I had available, and the tech workshop in the UC proceedings, I find that I am facing a gap in the documentation for the built-in processors.

Specifically, I wasn't able to find any info on the expression language used in the field calculator, or on any built-in functions that may be available e.g. for string manipulation, etc.

I do have two screen shots from the beta materials, one showing an arithmetic expression to convert units, the other showing two fields concatenated with a literal string. These both showed a JavaScript-style notation, but beyond that I would be guessing. Likewise, I don't seem to have anything for the Regular Expression processor - e.g. which regex notation is used, whether captures and back-references are supported, etc.

For one example, I was trying to figure out what expression syntax I could use to extract a substring from one field and concatenate that with the value of another field. I believe the functionality to do this is probably available in either of the built-in processors mentioned above, once I find out their respective expression formats.

Another example, if supported: changing a string value to all upper case. Yet another: rounding off a numeric value. I just don't know if there are built-in functions available in the Field Calculator to power these kinds of operations.

Perhaps someone who attended live training, or a member of the GeoEvent team can help out? Thanks!

[RJSunderman]

Bruce "?

Thank you for your questions. There is a product backlog for additional documentation "? specifically for how to configure the different types of Filter and Processor elements of a GeoEvent Service and the functionality exposed through these elements.

I will attempt to provide some examples below which address the questions you pose. You should be able to right-click any thumbnail image to open it in a new tab or window in order to view a larger version. The examles below are not exhaustive, merely illustrative. The Field Calculator, for example, supports all of the mathematic expressions found in the java.lang.Math class ... and we are looking to determine how best to expose string utilities from a similar Java library or class.

1)  Suppose you want to filter events whose field values do not match an expected pattern, such as to enforce an expectation that date values be expressed as YYYY-MM-DD. You can apply a Filter element configured with a MATCHES operation which will discard any events whose "?date"? values do not match the expected pattern. The pattern match below filters events whose "?date"? values only had a two digit year, were expressed MM/DD/YY, etc.

[ATTACH=CONFIG]26620[/ATTACH]

2)  Suppose that your "?date"? and "?time"? values arrive in separate fields and you wanted to concatenate them. You could couple a "?Field Calculator"? processor element with a "?Field Reducer"? to first concatenate the two values, overwriting either one with the result, then removing the now redundant left-over value. A field mapper could then be used to "cast"� the concatenated String value to a proper Date value.

[ATTACH=CONFIG]26621[/ATTACH]

Note that GEP expects date/time values in the ISO 8601 format (e.g. YYYY-MM-DDThh:mm:ss). However, input components using the out-of-the-box Text adapter can be configured with a mask to interpret string values which express date/time values in a different format (as long as the date/time value is ingested from a single field).

3)  The Introduction to GeoEvent Processor shows how a "?Field Calculator"? can be used to perform unit conversions using simple arithmetic expressions. The Java arithmetic operators are used to construct the expression. To convert kilometers to miles, multiply the value by a constant 0.621371 "� the result can be placed back into the input field, overwriting the original value, or appended to the event as a new value.

If the arithmetic expression were embedded in the round() function (one of the standard Math functions from the java.lang.Math class) you could round the value. The value could be manipulated to a number of significant decimals by multiplying and dividing by constant values as illustrated below:

[ATTACH=CONFIG]26622[/ATTACH]

4)  When using regular expressions, captures and back references are not supported. That is to say, you cannot match a string within a pattern and refer back to the matched string with \1. Suppose you had an isolated bit of XML and wanted to extract the value within a <tag>. You could use a series of regular expression matches to extract the value by first isolating the tagged value, then isolating the value with the single "?<"? and "?>"? on either side, then discarding the "?<"? and "?>"? delimiters "� each time replacing the value in the field with the value extracted from the field.

[ATTACH=CONFIG]26623[/ATTACH]

5)  If you wanted to extract data from one field and append the extracted sub-string to another field you would combine the operations from the previous example with the second example above (appending "?date"? and "?time"?). In this case you would append the extracted string value to the desired "?baseString"? field and then use a "?Field Reducer"? to remove the "?XML"? field which was no longer needed.

[ATTACH=CONFIG]26624[/ATTACH]

Here are a couple of points you"?ll want to remember. When appending strings, use single-quotes (not double-quotes) for literal strings in the expression. And when working with Regular Expressions, the field name upon which you are operating needs to be enclosed in curl-brackets with a "?$"? to designate it as a field (whereas everywhere else the field name is specified without quoting or bracketing the value).

<continued on next post...>

[RJSunderman]

<continued from last post...>

There are some known limitations when working with Filter and Processor elements in a GeoEvent Service. For one, when working with strings, tests for null and empty strings can be tricky. Connectors such as �??Watch a folder for new .csv files�?? and �??Receive text from a TCP Socket�?? represent missing data as empty strings when String values are expected. Empty strings are not null (they are simply zero-length strings). If you want to exclude events with attribute fields which are empty, use a MATCHES operation with the regular expression for an empty string and place the expression in a logical negation.

[ATTACH=CONFIG]26625[/ATTACH]

There is an enhancement in the product backlog for including additional string methods from a common library such as org.apache.commons.lang.StringUtils which will enable a �??Field Calculator�?? to leverage methods such as startsWith(), endsWith(), UpperCase(), IsAlpha(), and IsNumeric(). Look for this enhancement in a future release.

There is also a product backlog item for additional documentation �?? specifically for how to configure the different types of Filter and Processor elements of a GeoEvent Service and the functionality exposed through these elements.

Finally, the team is interested in feedback on both what is working for you and functionality you need in order to address the challenges you have when processing real-time data. Please visit the ArcGIS Ideas portal and remember to tag your ideas with �??GeoEvent Processor�?� and check the �??ArcGIS Server�?? box as a product category �?? this will help route your ideas to our team. Ideas and requested enhancements which receive a significant number of �??promotions�?� on the ArcGIS Ideas portal are more likely to be incorporated into the product backlog for inclusion in a future release than ideas posted to the forums.

Best Regards -
RJ