topic watching a folder with csv files. in ArcGIS GeoEvent Server Questions

watching a folder with csv files.

EliasLazarou — Fri, 07 Mar 2014 11:45:51 GMT

Hello there,

I am working on ArcGIS 10.2.1 with ArcGIS Server GeoEvent Processor. Actually i am trying to work with a csv file that is stored in a folder which i have already registered in the GeoEvent Processor. The scenario is the simplest i can do.

I create an input "Watch a folder for new .csv files" connector.

I create an output "Publish text to a tcp socket" connector.

I create a GeoEvent Service connecting the above.

Now i am supposed to open the command line and see the progress of the data feed in the port 5570 (the port set up for the output connector.), but i see nothing instead.

The csv file is not Read-Only.

Does anyone have a clue on that?

Thank you in advance,

Elias

Re: watching a folder with csv files.

RJSunderman — Fri, 07 Mar 2014 16:56:44 GMT

Hello Elias -

Let's try some simple checks first. On the GeoEvent Processor Manager's Monitor page, is the event count for the Watch a folder for new .csv files Input increasing? Keep in mind that if a GeoEvent Definition is not available to construct a GeoEvent from the data received - the comma separated text in this case - the event will be discarded. Your indication that this is happening is observing the event count for the Input is not increasing on the Monitor page. You might try allowing the Input to create an event definition and checking whether or not your Input is expecting the name of the event definition to be included as the 0th field of the event data. Also, keep in mind that the first two lines of the CSV file are reserved for a comma separated list of field name and a comma separated list of field data types - your actual event data needs to start with the 3rd line of the text file.

With no filtering or processing in-between the Input and Output, the event count 'In' and 'Out' for the GeoEvent Service should match. The GeoEvent Service receives the events 'In' and sends them 'Out' to the Publish text to a tcp socket Output.

In this case, there's nothing to debug with regard to the Output, as long as you were able to open the TCP Console application and see the message confirming that the app is listening for events on the designated port.

Have you tried stopping and restarting the the Watch a folder for new .csv files Input? This will clear the Input's buffers forcing a re-read of the CSV file. Please keep in mind that a known limitation of this Input is that once it has read data from a particular file, it will not re-read that file - even if the file's contents are changed. Deleting and dragging a new copy of the file into the folder being watched has no effect. The Input is remembering the file name and will not re-read the file to re-ingest its event data. You must stop/restart the Input in order to re-read the file, or change the file's name.

Hope this information helps -
RJ

Re: watching a folder with csv files.

MarkBramer — Fri, 07 Mar 2014 17:03:01 GMT

Hi Elias,

Does the .csv file already exist in the folder when your input, service and output are running? I've never used the watch-for-files inputs, but as I understand them, if a file already exists when the input is started, the input won't "hear" it. Certain file-related events (create, modify, save, etc) must happen for the watch-for-csv input to "hear" the file.

Try this: keep your existing scenario unchanged, and use Windows Explorer to copy-paste your file out of your registered folder and into another location, change the name, then copy-paste it back into your registered folder. I'm guessing GeoEvent Processor would "hear" the file creation and attempt to process it.

Here's some Java docs on watching file systems you may find useful: http://docs.oracle.com/javase/tutorial/essential/io/notification.html.

Hope this helps,
Mark

Re: watching a folder with csv files.

EliasLazarou — Wed, 12 Mar 2014 08:21:33 GMT

Thank you very much RJ and Mark.

Finally it worked by feeding the file to the folder after the service has been started.

Thanks again,

Elias

Re: watching a folder with csv files.

BrianBaldwin — Thu, 18 Aug 2016 22:35:16 GMT

Should maybe open a new thread, but the question is very closely related.

I have a folder with a CSV file that was 'found' by GeoEvent when it was first created, which created the GeoEvent Definitions. I have a python script that is updating the CSV on a regular basis (15 minute intervals), but the updated file is not discovered by GeoEvent. I tried to rectify this by setting GeoEvent to delete the input after a successful read, but the file is not deleted. Any ideas?

UPDATE: I also just added two lines in my python script to delete the file after a 5 second delay, but GeoEvent is not 'discovering' the creation of the new file either.

Re: watching a folder with csv files.

AlexanderBrown5 — Fri, 19 Aug 2016 13:31:27 GMT

Brian,

As RJ stated above regarding Watch a folder for new .csv files Input:

"Please keep in mind that a known limitation of this Input is that once it has read data from a particular file, it will not re-read that file - even if the file's contents are changed. Deleting and dragging a new copy of the file into the folder being watched has no effect. The Input is remembering the file name and will not re-read the file to re-ingest its event data. You must stop/restart the Input in order to re-read the file, or change the file's name."

Edit: 23-Jan-2018

Behavior for the ‘Watch a Folder for New CSV Files’ inbound connector was changed at 10.5.1 to no longer require that a file’s name be changed for the input to consider it a new file. The mechanism watching the folder for new files still does not consider file properties such as changes to a file’s “last updated” timestamp or file size. However, if you want an input to re-read files you’ve placed in a folder, you can simply stop and restart your input connector and each file’s content will re-read with its content processed as newly received event records.

Your python script needs to output an additional file into that folder, rather than update the existing file. Also, from RJ's response, you cant just add the same file back into the folder.

I would add some type of lookup in your python script to search the folder for the last filename, add a "_n" at the end of the new filename. If _1 exists, grab the integer and +1. For example your original file is "original.json", after your script runs it outputs an update to "original_1.json", next iteration would be "original_2.json". This would ensure the new file would get picked up by that type of input.

~Alex

Re: watching a folder with csv files.

BrianBaldwin — Fri, 19 Aug 2016 21:50:43 GMT

Alexander_Brown-esristaff‌ Thanks for pointing out that part of RJ's reply... definitely missed it.

Thanks for the idea/suggestion of creating a unique name on the file using the python script. My solution was to return a datetime.now and append that on to the end of the filename. I also kept the delete line in there so that I don't amass hundreds of old CSV files. It's working well though, thanks for the idea.

Re: watching a folder with csv files.

Han-WenLIU — Wed, 02 May 2018 08:21:02 GMT

I have the similar question:

I create an input "Watch a folder for new .csv files" connector.

than an output "Update a Feature"

and use a service to connect two of them.

In my monitor page I can see the input part is increasing

However, the service part and the output part remain 0

service

input

output

I have try to restart all of the input and output but still not working.

I think that in the input number is increasing means that my data can be read by geoevent right?

But how come the service not working?

Where do I miss??

kevinliu

Re: watching a folder with csv files.

BrianBaldwin — Wed, 02 May 2018 13:36:53 GMT

Add a screen shot of PTXTest, it is hard to provide an answer without knowing what the service looks like. Do you have a Field Mapper to 'join' the input to the output? The input is being read, but for whatever reason, your logic in the service is not getting to the output.

Re: watching a folder with csv files.

Han-WenLIU — Thu, 03 May 2018 02:16:02 GMT

Oh I'm sorry.

here is the screen shot:

and the mapper is like this

here are input and output

I have tried with or without field mapper, I can't get the data in both situation.

Re: watching a folder with csv files.

Than_HtetAung — Mon, 10 Sep 2018 08:44:26 GMT

Hi Guys,

I got the same issue, I still haven't got the input count even I change file name, add inside again.

I found an error in the log through.

Below is the error message, have any idea what are those?

esri.ges.fabric.internal.ZKPersistenceUtility

KeeperErrorCode = ConnectionLoss for /geoevent/config/clusters/default/arcgisServerConnections/b9fb6823-da12-4381-8e2b-855aa1aad9ea.json/ServicesCache.json org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /geoevent/config/clusters/default/arcgisServerConnections/b9fb6823-da12-4381-8e2b-855aa1aad9ea.json/ServicesCache.json at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)[23:org.apache.zookeeper.zookeeper-geoevent:3.5.0] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)[23:org.apache.zookeeper.zookeeper-geoevent:3.5.0] at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1908)[23:org.apache.zookeeper.zookeeper-geoevent:3.5.0] at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:272)[386:curator-framework:3.1.0] at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:268)[386:curator-framework:3.1.0] at org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:67)[387:curator-client:3.1.0] at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:99)[387:curator-client:3.1.0] at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:265)[386:curator-framework:3.1.0] at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:251)[386:curator-framework:3.1.0] at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:39)[386:curator-framework:3.1.0] at com.esri.ges.fabric.internal.ZKPersistenceUtility.writeToPath(ZKPersistenceUtility.java:286)[77:com.esri.ges.persistence.zookeeper.zk-persistenceutility:10.5.1] at Proxy3e3e16b3_7f0d_43ae_b629_760aa838293a.writeToPath(Unknown Source)[:] at Proxy7d6ab7f4_95f5_4a75_83ee_1f279b55660e.writeToPath(Unknown Source)[:] at com.esri.ges.persistence.zookeeper.agsconnection.ZKArcGISServerConnectionDAO.persistServicesInfo(ZKArcGISServerConnectionDAO.java:251)[87:com.esri.ges.persistence.zookeeper.agsconnection:10.5.1] at Proxy1b4537cc_78b0_4c5e_967f_95879f61bf6d.persistServicesInfo(Unknown Source)[:] at Proxy93a7a385_e4f7_4f07_a60c_10dce2c13549.persistServicesInfo(Unknown Source)[:] at com.esri.ges.manager.datastore.agsconnection.internal.ArcGISServerConnectionManagerImpl.onArcGISServerConnectionStatusChange(ArcGISServerConnectionManagerImpl.java:790)[217:com.esri.ges.manager.internal-agsconnectionmanager:10.5.1] at com.esri.ges.datastore.agsconnection.DefaultArcGISServerConnection$CacheUpdater.update(DefaultArcGISServerConnection.java:163)[109:com.esri.ges.framework.datastore.agsconnection-datastore:10.5.1] at com.esri.ges.datastore.agsconnection.DefaultArcGISServerConnection$CacheUpdater.run(DefaultArcGISServerConnection.java:199)[109:com.esri.ges.framework.datastore.agsconnection-datastore:10.5.1]

10 Sep 2018, 4:39:55 PM

ERROR

com.esri.ges.fabric.internal.ZKPersistenceUtility

10 Sep 2018, 4:33:25 PM

ERROR

Best Regards,

Than

Re: watching a folder with csv files.

wizgis — Thu, 02 Mar 2023 07:44:02 GMT

@Han-WenLIU @BrianBaldwin

I am also having a similar issue. Were you able to find a solution to it.

Re: watching a folder with csv files.

BrianBaldwin — Thu, 02 Mar 2023 13:50:08 GMT

@wizgis - This was years ago - but I vaguely remember inserting a 'date-time' at the end of my CSV file name when I was generating the files. That way - GeoEvent saw the CSV as a new file - even if after parsing, some of the records had not changed or updated, but it worked to ensure GeoEvent saw it as a new file each time.