Python Code Sample for tcp-text-in input?

[KeithFraley1]

Hi, I was wondering if anyone has an example writing to a tcp-text-in input in python?

To me, this is going to be the easiest and most efficient way to leverage data coming in from a variety of services.  Instead of writing a bunch of custom inputs, imo its easier to just to take a feed in python and write it out to a tcp-text-in input.

The question is do we have any python examples that can do that?  I have been playing around with my own but so far have had limited success.

[JacobBoyle]

I haven't done a text one, but I made one that outputs a JSON and you can use that.  you'll need to download the ODBC for python, pyodbc but the others are part of python.

import pyodbc
import json
import shutil
import collections
import time
import os

os.environ['TDSVER'] = '8.0'

connstr = 'DRIVER={SQL Server};SERVER=SERVERNAME;DATABASE=DATABASENAME;'
conn = pyodbc.connect(connstr)
cursor = conn.cursor()
 
cursor.execute(""" SELECT field1, field2, field3
            FROM TABLENAME            """)
 
rows = cursor.fetchall()
 
# Convert query to row arrays
 
rowarray_list = []
for row in rows:
    t = (row.field1, row.field2, row.field3)
    rowarray_list.append(t)
 
j = json.dumps(rowarray_list)
rowarrays_file = 'C:\Production\calculation\marcWells_rowarrays.js'
f = open(rowarrays_file,'w')
print >> f, j
 
# Convert query to objects of key-value pairs
 
objects_list = []
for row in rows:
    d = collections.OrderedDict()
    d['field1'] = row.field1
    d['field2'] = row.field2
    d['field3'] = row.field3
    objects_list.append(d)
 
j = json.dumps(objects_list)
objects_file = 'C:\Production\calculation\marcWells_objects.js'
f = open(objects_file,'w')
print >> f, j

date_string = time.strftime("%Y%m%d%H%M%S")

shutil.copyfile('C:\Production\input\marcWells_objects.js', r'C:\Production\input\marcWells_objects_' + date_string + '.js')


conn.close()

the last little bit above copies the last file generated and tags it with the time, then I run this script shortly after to provide the new JSON file.

mport shutil

shutil.copyfile('C:\Production\calculation\marcWells_objects.js', r'C:\Production\input\marcWells_objects.js')

conn.close()

[KeithFraley1]

Thanks for the quick response.

I was hoping for something a little more simple, I was thinking I could just write it out to a socket.  Has anyone attempted writing to a socket?


import socket

s = socket.socket()
s.connect((TCP_IP, TCP_PORT))
s.send(event)
s.close()

[GregYetman]

Hi,

I've been experimenting with this. The following two scripts (based on samples at python.org) will "talk" to each other. I haven't had the time to modify the server script to send data that GeoEvent will read, that's next up.

Hopefully this helps.

def main():
    import socket
    UDP_IP = '127.0.0.1'
    UDP_PORT = 5005

    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.bind((UDP_IP,UDP_PORT))

    while True:
        data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
        print 'received message:', data

if __name__ == '__main__':
    main()

def main():
    import socket
    UDP_IP = '127.0.0.1'
    UDP_PORT = 5005
    MESSAGE = 'Hi.'
    print 'UDP target IP:', UDP_IP
    print 'UDP target port:', UDP_PORT
    print 'Message:', MESSAGE

    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.sendto(MESSAGE, (UDP_IP,UDP_PORT))

if __name__ == '__main__':
    main()

[GrantTucker]

Thanks!  That latest sample did the trick.  I am now able to query a web service, write out the event to a socket and GEP picks it up and sends it to its proper location.

[ThomasGroßmann1]

Hi there

I tried gyetmans second script in order to talk to the GEP tcp input connector. The message is sent without errors. But GEP doesn't receive the message.:

import socket
UDP_IP = 'localhost'
UDP_PORT = 5565
MESSAGE = 'Vehicle,124,Truck,25,2012-07-24T08:11:00,"-117.192914,34.056459"'

print 'UDP target IP:', UDP_IP
print 'UDP target port:', UDP_PORT
print 'Message:', MESSAGE

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.sendto(MESSAGE, (UDP_IP,UDP_PORT))

Any ideas what could be the problem here?

Thanks
Thomas

[BillMajor]

Has anyone gotten this to work from Python?  Please provide an example.  I am trying to use the examples here and GEP never receives the messages, at least based on the Statistics Monitor.

[JamesCardona]

I use the code at the end of this post to send csv data to a GeoEvent Processor tcp text input.

When messages sent to a tcp-text input are not processed, I find a lot of the time that I have set the tcp-text input in GeoEvent Processor to expect a GeoEvent Definition in the incoming messages but the definition name is not the first attribute in each message?

import os, sys
import time, argparse
import socket

def sendData(sock, dataPath, interval):
    dataFile = None
    i = 0
    
    try:
        interval = float(interval / 1000.0)
        
        #open simulation data file and send data
        dataFile = open(dataPath, "r")
        print "Opened data file: %s" % dataPath
        for l in dataFile:
            sock.send(l)
            time.sleep(interval)
        dataFile.close()
        dataFile = None
        return True
        
    except Exception as e:
        raise Exception("Error sending data: %s" % e)
    finally:
        if dataFile:
            dataFile.close()
        
    
#make argument parser to handle user command line input
parser = argparse.ArgumentParser(description="Start a simulated data feed for a GeoEvent Processor tcp text input")
parser.add_argument("-d", "--data", help="path to file containing csv data. Default is ../data/sources/AsdiUal_sub.csv", default="../data/sources/AsdiUal_sub.csv")
parser.add_argument("-p", "--port", help="tcp socket port. Default is 5565", type=int, default=5565)
parser.add_argument("-n", "--name", help="host name of server to connect to. Default is 'localhost'", default="localhost")
parser.add_argument("-i", "--interval", help="interval in milliseconds between messages. Default is 500. Maximum interval is 5 seconds", type=int, default=500)


if __name__ == "__main__":
    args = parser.parse_args()
    if args.interval > 5000:
        print "Warning: interval cannot be more than 5 seconds. Setting interval to 5 seconds"
        args.interval = 5000
    if args.interval < 0:
        print "Warning: Invalid interval value. Setting interval to default value"
        args.interval = 1000    
    
    #begin simulation
    tcpSocket = None
    cont = True
    try:
        #Check if data file exists
        if not os.path.exists(args.data):
            raise Exception("Data file not found: %s." % args.data)
        
        #open socket
        tcpSocket = socket.create_connection((args.name, args.port))
        print "Opened tcp socket on %(name)s:%(port)s" % {"name": args.name, "port": args.port}
        
        while cont:
            sendData(tcpSocket, args.data, args.interval)
        
    except Exception as e:
        print "Error: %s" % e
    finally:
        if tcpSocket:
            tcpSocket.close()