topic Re: Writing to CSV issues in ArcGIS GeoEvent Server Questions

Writing to CSV issues

AndrewHargreaves — Thu, 25 Jun 2015 16:04:15 GMT

I have an output that writes to a CSV file. My Questions are:

I cannot figure out to get it to consistently overwrite the same CSV file - “example.csv” - each time it’s triggered.
How do I get it to NOT write out the FieldName before writing the value – I want only the value .

Thanks

Re: Writing to CSV issues

DanPatterson_Retired — Thu, 25 Jun 2015 17:01:51 GMT

Andrew, what are using to write the *.csv? Also, if something can't be deleted, it is probably some application is also using it (ie ArcMap, Windows Explorer etc)

Re: Writing to CSV issues

AndrewHargreaves — Thu, 25 Jun 2015 17:26:38 GMT

Dan

Attached is a screenshot of my GEE Service. My input detects a change of "status" on a Sewer overflow, if that status is "Verified" it strips out a bunch of unneeded data and pushes the remainder to a CSV file.

I have a python script that uses that CSV file to suck in the info and do some processing. I need that CSV file to always be named the same thing, and to only contain the value not the fieldname & value (in seperated fields) as it does now.

Thanks

Re: Writing to CSV issues

DanPatterson_Retired — Thu, 25 Jun 2015 17:37:19 GMT

I don't know what you are using to make the csv file, but if it is csv.writer, then you can omit the fieldnames. There are literally too many examples on the web about csv production with/and/without fieldnames for example

python - Pythonically add header to a csv file - Stack Overflow

13.1. csv — CSV File Reading and Writing — Python 2.7.10 documentation

I tend to use NumPy for that sort of thing...but you will stick with what you have and omit the first record. If you say it is writing a separate field for the field name and then the values in a separate column...that is a program logic problem and shouldn't be occuring

Re: Writing to CSV issues

AndrewHargreaves — Thu, 25 Jun 2015 17:41:30 GMT

Dan

I'm not using Python to create the CSV - It's a GEE output "Write to CSV file". I need to know if it's possible to configure this output to NOT have a date as part of the name GEE gives it when it creates it upon starting this output. I also need to know how to overwrite it each time the GEE service is triggered...

Re: Writing to CSV issues

DanPatterson_Retired — Thu, 25 Jun 2015 19:36:37 GMT

Apologies Andrew...I missing that oh so important detail that this was for the GeoEvent for server extension. Too many questions this week about *.csv files, overwriting and Python. I had a look through what I could find but to no avail, it appears the 'server' stuff is cloaked in a world parallel to conventional ArcGIS.

Re: Writing to CSV issues

AndrewHargreaves — Fri, 26 Jun 2015 11:46:41 GMT

Here is an answer from RJ Sunderman which indicates, sadly, I can't do what I want unless I hardcode the Filename into my Python script:

Hello Andrew –

I think there is some confusion in how the ‘Output Date Format’ and ‘Custom Date Format’ properties of a ‘Write to a CSV File’ output are used … and from where the “FieldName” to which you are referring is coming.

First, the ‘Write to a CSV File’ output uses the out-of-the-box Text adapter. This adapter’s implementation has some particular characteristics which a lot of folks find useful. The “FieldName” you are referring to, I think, is actually the name of the GeoEvent Definition associated with the GeoEvent delivered to the output.

For example, I have a GeoEvent Service which accepts event data sent to a GeoEvent hosted REST endpoint via HTTP/POST. The event data sent looks like this:

[

{

"TrackID": "AA-1234",

"TimeStamp": "3/15/2015 12:34:56",

"SensorCode": 11210

}

]

The GeoEvent Definition I created for my input to use, named GenericJsonReceiver, looks like this:

When I configure a ‘Write to a CSV File’ output I can leave the default ‘Output Date Format’ setting (‘ISO 8601 Format’) and I’ll get output in my named CSV file which looks like this:

GenericJsonReceiver,AA-1234,2015-03-15T12:34:56.000-07:00,11210

There are not field names in the data string. There is only the three data values prefixed with the name of the GeoEvent Definition. You cannot turn off this behavior. One of the features of the Text adapter is that it always prepends the name of the GeoEvent Definition associated with the event data. Precisely because you don’t have field names, seeing the name of the event definition can be very helpful. Also, this is one of the only outbound connectors you can use when you’re unsure which GeoEvent Definition is associated with your data once it is through the filtering and processing of a GeoEvent Service … and you want to find out.

If I change my ‘Write to a CSV File’ output configuration to specify a date/time mask, I can control what date/time value look like (e.g. how they are represented as String values) in the data written to the file. If I specify the ‘Output Date Format’ and ‘Custom Date Format’ properties as follows:

Then the text logged into the CSV file will appear as:

GenericJsonReceiver,AA-1234,15-Mar-2015 12:34:56,11210

What I’ve done is tell the output that I would like to see the date formatted as days, followed by a three letter abbreviation of the month, and a four digit year, then the time. I’ve also specified that I like dashed used in my date rather than slashes. These settings have absolutely nothing to do with the name of the output file.

The name of the output file is static, and configured up above the ‘Advanced’ parameters. You only get to specify a file prefix. Depending on how you elected to have the file rollover (the ‘File Rollover Method’ property) you will either get your specified file prefix with a .csv suffix … or you will get a system generated date/time stamp following your specified prefix with a .csv suffix. You have no control over how the transport (the out-of-the-box File transport in this case) decides to represent the date/time it integrates into the output file’s name.

Hope this information helps –

Re: Writing to CSV issues

DanPatterson_Retired — Fri, 26 Jun 2015 14:17:21 GMT

Thanks for the info Andrew