CSV in a Shared Folder

[AdamRepsher]

Hello Everyone,

I'm desperate at this point.  I am working at our state EMA supporting COVID-19 mapping.  I need to get GeoEvent a CSV file that is on a "Shared Network Drive".  I do not think that there is an input or a transport that includes passing a credential to connect, so I am seeking another way to deliver this CSV to GeoEvent.

I am going to list what is going on now so that someone sees a possible way through with what is being done so far:

• So far, I am getting port 445/TCP open on two firewalls between the two servers (for the next bullet)
• The other organization has shared their "drive" with MY enterprise user name.
• After the firewall rule is completed, here is the process:
  • I will make a Remote Desktop Connection to my GeoEvent Server.
  • I will Add a Network Connection to the other server's shared drive or folder.
  • I will then copy the CSV to a folder on my server.
  • I will then repeat the last step at least twice a day to grab new numbers.

From there, I will simply use the "Watch a Folder for New CSV Files" to grab changes with GeoEvent - when I over-write the local CSV with the new one.

Remember, I am a pure GIS person - not very skilled in networking, coding, etc.  Knowing that, read on...

I am thinking that if I had a "Windows Service" - running with the credentials of a Resource Account in our system - that could do the copy of the file on the other server to my server, I would be set.  Has anyone done this type of thing - or know of any other way that I could do it?

I wish I had enough knowledge to explain what inputs GeoEvent had to offer - and help the other server's IT staff see a way that they could throw something together that I could consume.

Real-Time & Big Data GIS‌

Thanks,

--Adam

[JakeSkinner]

Hey Adam,

Can the other organization share the folder to the ArcGIS Server account running ArcGIS Server/GeoEvent?  You can check what this account is by going to windows Services and look at the account specified for Log On As column for these services.

This would eliminate the need for the manual copy of the CSV file to the GeoEvent Server.  Instead, you could register the shared network drive as a Data Store in GeoEvent.

[Stefan_Jung · Accepted solution]

Hi Adam Repsher‌,

another option could be to create a batch script that will copy the file from the network share to a the local directory. In this case GeoEvent can watch this local folder. You can create a task in the Windows Task Scheduler to run this batch script ever x hours. 

The batch script could look like this: 

net use \\remoteserver\sharedfolder /user:domain1\user1 password
xcopy \\remoteserver\sharedfolder\file.csv C:\Temp\GeoEventWatch\ /Y
net use \\remoteserver\sharedfolder /delete‍‍‍

First of all you could test if this works like expected. It should do the same job that you already do manually.

Next step would be to use the Task Scheduler to run the script as often as you want.

Best,

Stefan

[AdamRepsher]

Thank you Stefan,

This is a big help.

--Adam

[EricIronside]

Hey Adam Repsher‌

Stefan P. Jung suggestion is a good one.  I would add the following additional step to your batch file:

W‌hen you copy the file from the romote server to your local drive, make sure you use a different extension than what GeoEvent is looking for. So assuming your geoevent is monitoring for '*.csv' files, you will want to xcopy the files as 'file.txt'. After the file is copied, add the following lines to your script to rename the file and include a date/timestamp in the name.

net use \\remoteserver\sharedfolder /user:domain1\user1 password
xcopy \\remoteserver\sharedfolder\file.txt C:\Temp\GeoEventWatch\ /Y

For /f "tokens=2-4 delims=/ " %%a in ('date /t') do (set mydate=%%c%%a%%b)
For /f "tokens=1-2 delims=/:" %%a in ("%TIME%") do (set mytime=%%a%%b)
rename c:\temp\GeoEventWatch\file.txt file_%mydate%_%mytime%.csv

net use \\remoteserver\sharedfolder /delete

Please see the following blog for more information https://community.esri.com/people/eironside-esristaff/blog/2019/04/11/geoevent-input-watch-a-folder-... 

Note: if you run the above commands outside of a batch file, change the '%%' to '%' in the "For ..." lines.

[RJSunderman]

Also consider that as you collect copies of data files you intend GeoEvent Server to read from, you probably want to configure the GeoEvent Server's input to delete files it has read after they have been processed. It is expected behavior at the 10.7.x release (and I think 10.6.x as well) that if you stop a running input used to read data from a file, any file(s) you have in the watched folder will be re-read when the input is restarted. Same goes for server machine reboot. You probably do not want GeoEvent to re-read files following a restart ... 

You will still need to append a date/time string to the file name, to ensure the external process used to copy files does not, at any point, try to lock or overwrite a file GeoEvent Server is reading. There is no way to coordinate GeoEvent Server's read (or file delete when done reading) with the external file copy process; filename uniqueness is how you work around that.

- RJ

[AdamRepsher]

RJ -  You are always there to make sure that I remember the finer points.  Thank you very much my friend.

--Adam

[AdamRepsher]

Fantastic addition!  Thank you!