Watch a folder csv problem deleting csv before fully processed

TylerKiehle1

I've got a problem with my workflow where OSIsoft's Pi Integrator for Business Analytics creates a .csv file and places it on a shared network directory.  Then GeoEvent Server has been configured with a "Watch folder for new .csv" input connector.  The connector is configured to delete file upon processing.  My problem is that the GeoEvent is deleting the file too quickly.  It appears the input connector begins processing the file as soon as it hits the folder and finishes processing before Pi has completed writing the data to the file.  So, I'm see sporadic errors in Pi Int for BA where the connection to the .csv is lost before it can complete writing the data to the file.

Anyone, run into this issue and resolved it?  I've attempted to modify the Batch Flush Interval from 500 millisec to 10000 millisec but it doesn't help.  Is there anyway to set this input connector to wait a period of time to begin processing after the file has dropped in the folder?

 

Thanks in advance,

Tyler 

watch_csv_config.JPG

KojiKUROSAWA_3-1732061263586.png

GregoryChristakos

Hi @TylerKiehle1 - There's no specific delay setting but there are two options you could consider:

1. Create a staging folder that GeoEvent doesn't look at. Once the file has been written entirely to the staging folder, you can use a script or something similar to move it to the folder that GeoEvent is looking at and will start reading from.

2. You can configure GeoEvent to look for a file of a specific name. Once Pi creates and writes all of its data to this file, use some sort of task/script to change the name to the specific one that GeoEvent is looking for.