Field Calculator help - just how numbers

Question

I have an incoming event definition which has a site name.  
This is the output from a TCP output connector:

Received Event : HITS,210142-US Kewell Ck,30/04/2014 00:15:00,0.672,0.0,á,á,á
Received Event : HITS,210137-Wangat,29/04/2014 23:00:00,0.268,18.0,á,15.5,á

In my featureclass, I only have a siteID, so for the above two, it would be 210142 and 210137.
This is always a 6 digit number.

Therefore, I figured this would be a good example of how a field calculator would be able to either* Trim the string to just show the first 6 characters
* Strip the string down to just show numbers (using regex)

I decided to go for the second option.

[ATTACH=CONFIG]33442[/ATTACH]

However, this just returns empty values:

Received Event : HITS,,29/04/2014 23:30:00,0.356,13.7,á,á,á
Received Event : HITS,,29/04/2014 23:45:00,0.668,42.520,á,á,á

I have tried various alternatives to the regex pattern, but it always gives blank values.  Is this because the incoming event is a string and therefore there is no numbers?

So, in priority order:
* How would you solve this with approach #2
* How would you solve it with approach #1
* Bonus points for helping me convert those incoming blank values which are coming in from the input feed as \u00a0 and ending up being á.  Not sure what happens when I throw these at a double field in my feature class, might not matter.

FieldCalc1.jpg

SimonJackson · Answer

RJ - ask Adam for a pay rise.
Both your posts have cleaned up my output feed perfectly.
I think I will end up making this public, so ill post back in a week or so with a Javascript Operations Dashboard link

RJSunderman · Answer

Simon -

Polling the feed you indicated:  [url=http://pipes.yahoo.com/pipes/pipe.run?_id=7892c39c1778e81c27aeb4a0ba265623&_render=json]http://pipes.yahoo.com/pipes/pipe.run?_id=...&_render=json[/url] it appears to me that all of the items event fields are being expressed as JSON strings, with the first item in the list advertising the expected field names. 
:
:
items: [
{
Site: "Site",
LastSampleTime: "Last Sample Time",
RiverLevel: "River Level",
FlowRate: "Flow Rate",
RainfallIntensity: "Rainfall Intensity",
WaterTemp: "Water Temperature",
Conductivity: "Conductivity",
description: null,
title: null
},
{
Site: "210001-Singleton",
LastSampleTime: "30/04/2014 05:57:56",
RiverLevel: "0.652",
FlowRate: "400.2",
RainfallIntensity: " ",
WaterTemp: " ",
Conductivity: " ",
description: null,
title: null
}, ...

The GeoEvent Definition generated for me by the GeoEvent Processor input specifies that all fields are expected as String:
[ATTACH=CONFIG]33454[/ATTACH]

When I send the poll'd JSON to a GeoEvent Cache, I can review the event data as either Text or JSON by visiting the appropriate REST endpoint for the cache:* localhost:6180/geoevent/rest/cache/cache-out?f=text
* localhost:6180/geoevent/rest/cache/cache-out?f=generic-json

The GeoEventCache adapter used by the Publish GeoEvents on a REST endpoint appears to be handling the unicode string/character \u00a0 being reported when values are missing/unreported as part of the unicode extended character set.
[ATTACH=CONFIG]33455[/ATTACH]

The Text adapter used by the Publish text to a TCP Socket, on the other hand, appears to be converting the "non breaking space" character ... which in HTML is &# 160; ... to an ASCII extended character equivalent:  á.

What you could do, to avoid having this unicode character being interpreted by different adapters, is edit the generated GeoEvent Definition to specify that the metric values are expected as type Double. The input will then attempt to cast the JSON string values it receives to Double, and set the field's value to null if the type cast fails. Might as well include the description and title event fields from the provided JSON, currently being reported as null strings, while you're at it:
[ATTACH=CONFIG]33457[/ATTACH]

Notice in the following illustration that in a "text" adaption of the event data, the null fields appear "empty", while in a JSON adaption of the event data the null fields are not reported.
[ATTACH=CONFIG]33459[/ATTACH]

I think this will better allow you to apply filter criteria, process events, and eventually send event data to a feature service. Better, in my opinion, to send "null" data as null than as a "non-breaking space" unicode character.

- RJ

Test-JSON-UnicodeCharacters (2).png

UnicodeCharacters-Cache-Output.png

104815UnicodeCharacters-Cache-Output.png

Test-JSON-UnicodeCharacters.png

RJSunderman · Answer

Hello Simon -

I think the problem is with the Field Calculator (Regular Expression) processor's configuration.

Looking at your screenshot, the Field Name has been specified as:  Site.  When using the "regular" Field Calculator processor, field names are expressed without decoration. When using the Field Calculator (Regular Expression) processor you need to enclose field names in curl-braces and prepend a '$' ... so your configuration needs to specify ${Site} rather than just Site.

This is a common source of confusion. With the 10.2.2 product release the "regular" Field Calculator processor will accept Java String functions - some of which accept RegEx expressions. Hopefully this will reduce the need to use the Field Calculator (Regular Expression) processor ... and help eliminate the confusion in how the two processors expect event field names be specified.

For example, if you are using 10.2.2 you could configure a "regular" Field Calculator processor with an expression like replaceAll(fieldName,'[-].*','') to identify and replace all characters following a literal '-' with an empty string. That would effectively trim SiteId-SiteName expression down to just the 6-digit SiteId.  (You could also use the substring() function to grab just the first six characters of the SiteId-SiteName expression ... something like substring(fieldName,0,6).)

If you are using 10.2.1 you won't have support for Java String functions. In that case you'll have to stick with the Field Calculator (Regular Expression) processor, specify the field name ${Site}, use a RegEx pattern like ^[0-9][0-9]*, and either place the substring into a new field (allowing you to specify that it is an Integer or Long) ... or overwrite the existing Site field value with the substring as a string representation of the SiteId.