topic Re: se_toolkit - Control File Documentation in Data Management Questions

se_toolkit - Control File Documentation

bbieb — Mon, 08 Aug 2011 15:57:56 GMT

Is there any documentation on creating a control files that are used with the se_toolkit? For example using the control file with the asc2sde tool? ftp://ftp.esri.com/pub/staff/vangelo/se_toolkit/docs/dat/asc2sde.html
Thanks.
brian

Re: se_toolkit - Control File Documentation

VinceAngelo — Mon, 08 Aug 2011 16:31:13 GMT

I've had an overhaul of the DAT control file documentation on my "to do" list for a while now,
but the time to accomplish it has been sparse. There is a lot of information in the old Forums
(forums.esri.com), with details on specific applications, but samples/geonames.ctl and the
text provided in the existing asc2de, ascinfo, and sdeupdate documentation are all that is
provided with an install (besides the source tree, of course).

What in particular are you trying to accomplish?

- V

Re: se_toolkit - Control File Documentation

bbieb — Mon, 08 Aug 2011 17:03:21 GMT

I had the live feeds running on my sde (http://www.arcgis.com/home/item.html?id=9ff56fbe3c4f4b749fd6066424685280) this winter and when the NWS changed their us.cap file, the Weather Warning/Watches feed broke. I am finally getting around to fixing it and there needs to be quite a bit more parsing of string fields to get the data out of the csv file. What I really need is the string functions that can be utilized in the control file. Hopefully I can figure it out from there.
SUBSTR - have an example, but what else is there?

Re: se_toolkit - Control File Documentation

VinceAngelo — Mon, 08 Aug 2011 17:24:32 GMT

I added a CSVMODE table tag to simplify CSV parsing, though I didn't spend a whole lot
of time trying to break it.

You'll probably want to look a the SubToken class (in dat_string.c), which breaks strings
with arbitrary delimiters (e.g., '1|2|a:b:c|3').

I'm doing release QA for setk10.0 at the moment, and hope to have it published soon.

- V

Re: se_toolkit - Control File Documentation

bbieb — Mon, 08 Aug 2011 20:36:05 GMT

Hi Vince,
Thanks for your prompt replies. I've been looking at your code but haven't been able to decipher how to split or trim the following:
Severe Thunderstorm Warning issued August 08 at 4:59PM EDT expiring August 08 at 5:30PM EDT by NWS Tallahassee http://www.srh.noaa.gov/tlh/

I need to parse out "Severe Thunderstorm Warning", "August 08 at 4:59PM EDT" and "August 08 at 5:30PM EDT". Tried using SubToken but if I use "issued" for the delimiter, it uses "i", "s", etc for delimiters.

Re: se_toolkit - Control File Documentation

VinceAngelo — Fri, 10 Dec 2021 23:07:17 GMT

The SubToken delimiter is character oriented -- "issued" becomes "i" or "s" or "u" or "e" or "d".

Do you have 'sed' available?

 
echo "Severe Thunderstorm Warning issued August 08 at 4:59PM EDT expiring August 08 at 5:30PM EDT by NWS Tallahassee http://www.srh.noaa.gov/tlh/" | \
sed 's/ issued /|/' | \
sed 's/ expiring /|/' | \
sed 's/ by /|/' | \
sed 's/ http/|http/'

- V

Re: se_toolkit - Control File Documentation

VinceAngelo — Fri, 10 Dec 2021 23:07:19 GMT

I'm working on a solution to address this parsing issue inside se_toolkit -- I'm adding a
"StrPos" class:

 
 StrPos(strcol,"string"[,{START | END | BEFORE | AFTER}])
StrIPos(strcol,"string"[,{START | END | BEFORE | AFTER}])

With it you could use [this would be more legible with a fixed-width font]:

 
event         String                            - 200 N
$stop1        StrPos(event," issued ",BEFORE)   -   4 N
$start2       StrPos(event," issued ",AFTER)    -   4 N
$stop2        StrPos(event," expiring ",BEFORE) -   4 N
$start3       StrPos(event," expiring ",AFTER)  -   4 N
$stop3        StrPos(event," by ",BEFORE)       -   4 N
event_class   SubStr(event,1,$top1)             -  80 N
event_issued  SubStr(event,$start2,$stop2)      -  80 N
event_expires SubStr(event,$start3,$stop3)      -  80 N

Use the "Contact Author" page from ArcScripts to send me your contact info, ArcSDE release,
and platform, and I'll get you a "setk93b41" snapshot with the StrPos class (setk100 will include
this at general release).

- V

Re: se_toolkit - Control File Documentation

VinceAngelo — Fri, 10 Dec 2021 23:07:22 GMT

Ack. The above won't work, since SubStr is {col,start,length}, but this should:

 
event         String                            - 200 N
$stop1        StrPos(event," issued ",BEFORE)   -   4 N
$start2       StrPos(event," issued ",AFTER)    -   4 N
$sub1         Substr(event,$start2)             - 200 N    
$stop2        StrPos($sub1," expiring ",BEFORE) -   4 N
$start3       StrPos($sub1," expiring ",AFTER)  -   4 N
$sub2         Substr($sub1,$start3)             - 200 N  
$stop3        StrPos($sub2," by ",BEFORE)       -   4 N
event_class   SubStr(event,1,$stop1)            -  80 N
event_issued  SubStr($sub1,1,$stop2)            -  80 N
event_expires SubStr($sub2,1,$stop3)            -  80 N

- V

Re: se_toolkit - Control File Documentation

VinceAngelo — Tue, 23 Aug 2011 23:10:29 GMT

The StrPos and SubStr classes have been updated to support the above usage, and a
full setk93b41 release is now available at the FTP site. The next release of 'se_toolkit'
will be setk100.

- V

Re: se_toolkit - Control File Documentation

bbieb — Fri, 26 Aug 2011 14:39:43 GMT

Thanks Vince, it works as advertised!
brian