GeoEvent 10.6.1 crushing CPU

[AllenScully]

Upgraded ArcGISServer + Geoevent on our test GeoEvent machine from 10.5. to 10.6.1:

(Server 2012 R2,  Virtual, 2 CPU + 16 GB RAM)

CPU usage is basically pegged, constantly.  I was eventually able to get into GeoEvent manager and stop all services (they all need attention - 10.6.1 GeoTagging seems to be a bit different) - but even with all Services/Inputs/Outputs stopped, CPU usage is spiked.  - 

In addition to the expected java.exe and javaw.exe, there are 2 erl.exe processes contributing to the situation.  Have rebooted, re-started services - cannot get CPU to come down, so the whole server is functionally down for now.  

Only a few ArcServer services running, so I'm looking to learn if something was done incorrectly on install, or if we just need to bump up CPU for GeoEvent 10.6.1?

Thanks - 

Allen

[AdamRepsher]

Allen Scully‌,

I read this, and two things came to mind:

1. Are you running any McAfee products on the server?  If so, look at these: 
   https://support.esri.com/en/technical-article/000014907
   https://support.esri.com/en/technical-article/000012517 
   I had this problem at 10.5 or 10.5.1 as I remember.  Mine was fixed with a patch of the Enterprise McAfee that was pushed out to all servers.  IT would not whitelist specific folders for me.
2. Running GeoEvent with 2 CPUs in a virtual environment. 

Not really convinced that either of these are contributing especially since this only occurred after updating from 10.5 to 10.6.1.  It is not like you decreased CPU count or memory from a previous install.  That said, I have never done a 10.5 to 10.6.1 update and I have always needed to get a bit of help from Esri support after an install or update.

Just shot-gunning some ideas.  I hope you find a solution quickly,

--Adam

[AllenScully]

Thanks Adam - 

No McAfee stuff running - we use a product called Cylance, which if it doesn't block an application, is pretty unobtrusive.  

I do think 2 CPU is on the low side since GeoEvent is a hog, but for our test site, we don't have much running currently and I actually stopped all of the services and inputs in GEP. 

Nothing else running on this machine other than Server/GEP.  

After 48 hours or so of just ignoring the server, it's not sitting at 95%+ but is still very erratic - down to 18% one minute then ramping up to 95% +, never staying at either high or low usage for too long.  

Will follow up with what I find for a solution.

Allen

[AdamRepsher]

Hi Again!

So, things have changed for me.  For the past two weeks, I have been attempting to build a complex service to reformat incoming AVL packets from one non-standard schema to another.  It's a pain in the rear, as companies should realize that when they do this - they make it so complex for their customer unnecessarily - ESPECIALLY since the company never makes ANYTHING that consumes these packets!!!!  Standards, Please!!!!  Enough of my rant....

So, doing all of this conversion of every field needed - after every step, I attach a JSON or CSV file output to make sure that everything runs as expected.  After doing this successfully for around a 51 step process, branched by IF statements, the server runs smoothly.  Averaging CPU usage between high teens and low 20s when running. 

I then attach a Text to External UDP Socket - and the server spikes to 99% or 100% CPU usage. 

Of note, this process works flawlessly in a GE at 10.3.1.  I did not export this one and use it in my current server.  I have rebuilt this service 3 times in 10.6.1 and have re-created outputs numerous times.  No luck.  I do have a few recurring error messages in the log that I have to figure out.