Watch a folder csv problem deleting csv before fully processed

[TylerKiehle1]

I've got a problem with my workflow where OSIsoft's Pi Integrator for Business Analytics creates a .csv file and places it on a shared network directory.  Then GeoEvent Server has been configured with a "Watch folder for new .csv" input connector.  The connector is configured to delete file upon processing.  My problem is that the GeoEvent is deleting the file too quickly.  It appears the input connector begins processing the file as soon as it hits the folder and finishes processing before Pi has completed writing the data to the file.  So, I'm see sporadic errors in Pi Int for BA where the connection to the .csv is lost before it can complete writing the data to the file.

Anyone, run into this issue and resolved it?  I've attempted to modify the Batch Flush Interval from 500 millisec to 10000 millisec but it doesn't help.  Is there anyway to set this input connector to wait a period of time to begin processing after the file has dropped in the folder?

Thanks in advance,

Tyler 

[GregoryChristakos]

Hi @TylerKiehle1 - There's no specific delay setting but there are two options you could consider:

1. Create a staging folder that GeoEvent doesn't look at. Once the file has been written entirely to the staging folder, you can use a script or something similar to move it to the folder that GeoEvent is looking at and will start reading from.

2. You can configure GeoEvent to look for a file of a specific name. Once Pi creates and writes all of its data to this file, use some sort of task/script to change the name to the specific one that GeoEvent is looking for.

[TylerKiehle1]

Thank you for your response.  That is a good suggestion, but I was hoping that Esri's configurable solution had addressed this shortcoming and that I wouldn't have to script a way around it.  I guess I could potentially take the time to learn how to build my own input connector that is the exact same, except with a delay setting so it works as intended.

[JeffSilberberg]

@TylerKiehle1 

     What you describe is not a geoEvent issue it's a known working constraint of this kind of interface.  No matter what the application, you should always Move the completed File to the directory/mount point being monitored not create it there.   This is because the first of a number of system events will fire on the Open(). 

[TylerKiehle1]

Thank you for your response.  Be that as it may, if it's a known issue, it would seem that either the Pi Integrator for BA or GeoEvent could program their solutions to take it into account.  The GeoEvent input connector could have a setting that would allow for a delay between recognizing a file has landed in the folder and opening the file for processing.  If I can script a solution, they can design it as a function of their configurable solution.  After all, I'm pretty sure Esri is going for ease of use with configurable solutions, this would be one of those value added features that would help save end users time and resources.