topic Re: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms. in ArcGIS GeoEvent Server Questions

GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

ThorstenBraun — Wed, 29 Jul 2015 11:05:28 GMT

Maybe someone has an idea to find a workaround or solution for that problem:

The GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input
receives more than a message in ~ 100 ms. The Input geoevents are not dropped like it
appears to be in 10.2.2, instead the Input/TCP-Port stopps working at all.

Even restarting the Input Connector does not solve the problem. If the
Input connector stopped working once, no more data is received until the
complete GeoEvent Service has been restarted.

We had problems with remarkable packet loss in 10.2.2, so we decieded to use 10.3.1.

Now we are running into the problem described above that the incoming data is not processed

if the input stopped working once.

We reproduced the problem with a simple python tool that is sending JSON Data to the TCP Socket.

The test script sends always the identical JSON document. The JSON is about 1000 characters long.

The GeoEvent Service always processes some Input Events (5 to 20, it's always different), then the

Input stops working as described above.

Test Environment:

Windows: Server 2012 R2 and Windows 7 Professional
Running on a Hyper-V Virtual Machine with 16GB RAM and 4 Cores
Tested on 3 different Machines (one with Portal federated the others blank ArcGIS Server and GEP)
ArcGIS Versions 10.3.1
GEP Connector: TCP JSON IN

The most Important aspect is, that the Input connector does not recover itself, instead it is totally
broken until the complete GEP is restarted. There is no chance to evaluate more
detailed where the problem could be. We did not find anything helpfull in the Logfiles.

It would be great if we could get some assistence or any hint what exactly the problem is and maybe how

to circumvent the Problem.

Unfortunately we are limited to TCP/JSON as Input, because this is the Interface to our customer.

Re: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

GianlucaCaporossi — Wed, 29 Jul 2015 12:42:32 GMT

for your information

we use tcp / geomessages with peaks of more than 300 tracks per second without any problem.

Windows: Server 2008 R2

Running: on Amazon cloud machine with 8GB RAM and 2 Cores

ArcGIS: Versions 10.3.1

The tests are made using a machine for sending tracks and one for receiving the geoevent.

The use of a single machine (sending and receiving) slows down performance by far.

I hope this information will be useful for you.

if you want I can test your file on my server.

Ciao

Gianluca

Re: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

ThorstenBraun — Wed, 29 Jul 2015 13:59:39 GMT

Hi,

I'm using the following configuration for the geo Event Input:

# Python script to reproduce the Problem.

import socket
import json
import time

# JSON Message that will be send as single object

for j in range(10):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print "connecting.." + str(j)
s.connect(('127.0.0.1', 15057)) #

#sending the message 20 times
for i in range(1,21):
    print "send data.. " + str(i*j)
    s.send(json.dumps([{'sampleString': '1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678902345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' + str(i*j)}]))
    time.sleep(0.1) # remove this line to break down geo event input

s.shutdown(1)
s.close()

Exit

Re: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

GianlucaCaporossi — Wed, 29 Jul 2015 15:15:00 GMT

I tested on my machine

in: genericjson / tcp

Out: out-csv file

remove the line time.sleep (0.1)

and sent 10,000 json

everything works without problems

Re: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

ThorstenBraun — Wed, 29 Jul 2015 15:22:28 GMT

My Input was configured like this:

Re: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

GianlucaCaporossi — Wed, 29 Jul 2015 15:42:13 GMT

Re: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

ThorstenBraun — Thu, 30 Jul 2015 09:22:18 GMT

Hi,

strange. In my opintion there must be a difference somehow. We were able to reproduce this on several machines.

- Which operating system are you testing on?

- What is the locale of your OS?

- Are you sending to localhost or from another machine to your geoevent Service?

- Do you have any special configuration for geoevent extension?

- Is there a patch for 10.3.1 which I didn't notice yet?

# here is another script which breaks down the Input.

# -*- coding: cp1252 -*-
import socket
import json
import time

#sending the message 20 times
for i in range(100):
   s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
   s.connect(('127.0.0.1', 15057)) #

   j = 0
   n = 3
   while j < n:
      print "send data.. " + str(i)
      s.send(json.dumps({'sampleString': '123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' + str(i*10+j)}))
      j = j + 1


   s.shutdown(1)
   s.close()
   #time.sleep(1)

exit

Re: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

GianlucaCaporossi — Thu, 30 Jul 2015 14:52:15 GMT

- Which operating system are you testing on? Windows server 2008 R2 Service Pack 1

- What is the locale of your OS? Cloud Amazon

- Are you sending to localhost or from another machine to your geoevent Service? I tested in both ways

- Do you have any special configuration for geoevent extension? NO

- Is there a patch for 10.3.1 which I didn't notice yet? NO

I also tested the new script that you sent me and I can confirm that it works without problems

Attached are the file export of input,export and service used.

Re: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

XuehanJing — Fri, 31 Jul 2015 23:54:12 GMT

Hello all,

Same issue has been tested and reproduced in Technical support. We have already logged bug for it. Out development team also reproduced the issue and was currently working on this issue. The bug id BUG-000 89145 is for your reference.

Thanks.

Re: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

XuehanJing — Fri, 31 Jul 2015 23:54:57 GMT

Sorry the bug number is BUG-000089145.

Re: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

ThorstenBraun — Mon, 03 Aug 2015 06:54:38 GMT

Hi Xuehan,

thanks for your Information.

Do you have an idea why the script is working on Gianluca's system?

Is there already an information what exactly the problem is?

Is it a timing issue, problem with JSON parsing or a problem with the TCP socket?

If there won't be a patch for the problem soon we are forced to find a workaround that.

This could be e.g. the development of a custom Input connector.

Therefore it is important for us to know, whether this is a generic issue, which can not be

solved by a custom Input Adapter or whether it is a bug specific for that TCP/JSON Adapter?

The best solution anyhow would be patch for this.

Regards,

Thorsten

Re: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

XuehanJing — Mon, 03 Aug 2015 19:20:35 GMT

Hi Thorsten,

Currently, I don't have additional information for most of your question. But the thing I can tell for sure, is that our development team will work to solve this bug in the future release, since the bug status has been changed to "Assigned". The bug information is also public at this time in BUG-000089145 - The JSON-tcp-in input connector stops working if t..

Thanks.

Re: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

GianlucaCaporossi — Sun, 12 Dec 2021 03:29:46 GMT

Hi Thorsten,

I did some more tests,

the problem is not in the number of json but in the number of TCP connections.

If you open one connection tcp and sends thousand of json there are no problems.

# here is another script which breaks down the Input.

# -*- coding: cp1252 -*-
import socket
import json
import time

#sending the message 1000 times on 1 socket
for i in range(1):
   s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
   s.connect(('127.0.0.1', 4199)) # 
   
   j = 0
   n = 1000
   while j < n:
      print "send data.. " + str(i)
      s.send(json.dumps({'sampleString': '123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' + str(i*10+j)}))   
      j = j + 1
   
   javascript:;
   s.shutdown(1)
   s.close()
   #time.sleep(1)  
exit

Opening multiple connections and behavior depends on the version of Windows server (web, standard enterprise, data center), you are using.

The machine on Amazon on which I made the first tests, is a DataCenter and probably for this reason I was having problems because it supports a higher number of TCP connections.

I redid the tests on a vm on my laptop with Windows 2008 R2 Standard and I had your problems.

I tweaked tcp connections modify the Windows Registry with the details taken here (http://kb.globalscape.com/KnowledgebaseArticle10438.aspx)

I put the tcp.reg file attached but you have to modify the GUI of your network cards.

Also you should use ports below 5000

so up to 500 TCP connections not finding problems, clearly open other connections we return to the problem than before windows releases the tcp connection.

# here is another script which breaks down the Input.

# -*- coding: cp1252 -*-
import socket
import json
import time

#sending the message 1500 times on 500 socket
for i in range(500):
   s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
   s.connect(('127.0.0.1', 4199)) # 
   
   j = 0
   n = 3
   while j < n:
      print "send data.. " + str(i)
      s.send(json.dumps({'sampleString': '123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' + str(i*10+j)}))   
      j = j + 1
   
   
   s.shutdown(1)
   s.close()
   #time.sleep(1)  
exit

I hope this info can be helpful for you to find a workaround.

ciao

Gianluca

Re: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

ThorstenBraun — Tue, 04 Aug 2015 09:22:36 GMT

Hi Gianluca,

thanks for doing some more testing on this issue.

I did some tests on a Windows Server 2012 R2 machine with 10.3.1.

In my configuration there is only one TCP Server socket configured in GeoEvent Extension.

I used the first script above, which opens the socket once.

My Input connector did not process any message.

Regards,

Thorsten

Re: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one message per 100ms.

TimothyStiles — Thu, 17 Sep 2015 20:08:28 GMT

Changes made for the 10.4 release appear to address this issue. A potential fix for the 10.3.1 release was delivered to a customer for verification. Other customers requiring this fix for the 10.3.1 release should work with their Technical Account Manager to submit a hot fix request. Please keep in mind that hot fixes are targeted releases and may not be applicable in all customer environments.