GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

5690
14
Jump to solution
07-29-2015 04:05 AM
ThorstenBraun
New Contributor II

Maybe someone has an idea to find a workaround or solution for that problem:


The GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input
receives more than a message in ~ 100 ms. The Input geoevents are not dropped like it
appears to be in 10.2.2, instead the Input/TCP-Port stopps working at all.

Even restarting the Input Connector does not solve the problem. If the
Input connector stopped working once, no more data is received until the
complete GeoEvent Service has been restarted.

We had problems with remarkable packet loss in 10.2.2, so we decieded to use 10.3.1.

Now we are running into the problem described above that the incoming data is not processed

if the input stopped working once.

We reproduced the problem with a simple python tool that is sending JSON Data to the TCP Socket.

The test script sends always the identical JSON document. The JSON is about 1000 characters long.

The GeoEvent Service always processes some Input Events (5 to 20, it's always different), then the

Input stops working as described above.

Test Environment:

  • Windows:      Server 2012 R2 and Windows 7 Professional
  • Running         on a Hyper-V Virtual Machine with 16GB RAM and 4 Cores
  • Tested           on 3 different Machines (one with Portal federated the others blank ArcGIS Server and GEP)
  • ArcGIS           Versions 10.3.1
  • GEP               Connector: TCP JSON IN

The most Important aspect is, that the Input connector does not recover itself, instead it is totally
broken until the complete GEP is restarted. There is no chance to evaluate more
detailed where the problem could be. We did not find anything helpfull in the Logfiles.

It would be great if we could get some assistence or any hint what exactly the problem is and maybe how

to circumvent the Problem.

Unfortunately we are limited to TCP/JSON as Input, because this is the Interface to our customer.

14 Replies
ThorstenBraun
New Contributor II

Hi Xuehan,

thanks for your Information.

Do you have an idea why the script is working on Gianluca's system?

Is there already an information what exactly the problem is?

Is it a timing issue, problem with JSON parsing or a problem with the TCP socket?

If there won't be a patch for the problem soon we are forced to find a workaround that.

This could be e.g. the development of a custom Input connector.

Therefore it is important for us to know, whether this is a generic issue, which can not be

solved by a custom Input Adapter or whether it is a bug specific for that TCP/JSON Adapter?

The best solution anyhow would be patch for this.

Regards,

Thorsten

0 Kudos
XuehanJing
New Contributor III

Hi Thorsten,

Currently, I don't have additional information for most of your question. But the thing I can tell for sure, is that our development team will work to solve this bug in the future release, since the bug status has been changed to "Assigned". The bug information is also public at this time in BUG-000089145 - The JSON-tcp-in input connector stops working if t..

Thanks.

0 Kudos
GianlucaCaporossi
New Contributor II

Hi Thorsten,

I did some more tests,

the problem is not in the number of json but in the number of TCP connections.

If you open one connection tcp and sends thousand of json there are no problems.

# here is another script which breaks down the Input.

# -*- coding: cp1252 -*-
import socket
import json
import time

#sending the message 1000 times on 1 socket
for i in range(1):
   s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
   s.connect(('127.0.0.1', 4199)) # 
   
   j = 0
   n = 1000
   while j < n:
      print "send data.. " + str(i)
      s.send(json.dumps({'sampleString': '123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' + str(i*10+j)}))   
      j = j + 1
   
   javascript:;
   s.shutdown(1)
   s.close()
   #time.sleep(1)  
exit

Opening multiple connections and behavior depends on the version of Windows server (web, standard enterprise, data center), you are using.

The machine on Amazon on which I made the first tests, is a DataCenter  and probably for this reason I was having problems because it supports a higher number of TCP connections.

I redid the tests on a vm on my laptop with Windows 2008 R2 Standard and I had your problems.

I tweaked tcp connections modify the Windows Registry with the details taken here (http://kb.globalscape.com/KnowledgebaseArticle10438.aspx)

I put the tcp.reg file attached but you have to modify the GUI of your network cards.

Also you should use ports below 5000

so up to 500 TCP connections not finding problems, clearly open other connections we return to the problem than before windows releases the tcp connection.

# here is another script which breaks down the Input.

# -*- coding: cp1252 -*-
import socket
import json
import time

#sending the message 1500 times on 500 socket
for i in range(500):
   s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
   s.connect(('127.0.0.1', 4199)) # 
   
   j = 0
   n = 3
   while j < n:
      print "send data.. " + str(i)
      s.send(json.dumps({'sampleString': '123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' + str(i*10+j)}))   
      j = j + 1
   
   
   s.shutdown(1)
   s.close()
   #time.sleep(1)  
exit

I hope this info can be helpful for you to find a workaround.

ciao

Gianluca

0 Kudos
ThorstenBraun
New Contributor II

Hi Gianluca,

thanks for doing some more testing on this issue.

I did some tests on a Windows Server 2012 R2 machine with 10.3.1.

In my configuration there is only one TCP Server socket configured in GeoEvent Extension.

I used the first script above, which opens the socket once.

My Input connector did not process any message.

Regards,

Thorsten

0 Kudos
TimothyStiles
New Contributor III

Changes made for the 10.4 release appear to address this issue. A potential fix for the 10.3.1 release was delivered to a customer for verification. Other customers requiring this fix for the 10.3.1 release should work with their Technical Account Manager to submit a hot fix request. Please keep in mind that hot fixes are targeted releases and may not be applicable in all customer environments.

0 Kudos