Portal for ArcGIS 10.5.1: Extremely Slow Performance

2188
5
Jump to solution
01-29-2019 01:41 PM
SamJehle1
New Contributor III

I'm currently trying to troubleshoot an issue with an internal, highly available Portal environment and have hit a wall. After some weekend patching on our virtual machines hosting Portal on 1/26, the performance for Portal completely tanks and it doesn't matter which part of this site. Web applications take around 10-15 minutes to load even. When even trying to hit the "../arcgis/sharing/rest" endpoint it takes upwards of 10 seconds, which did not happen before the patching.There only seems to be an issue with Portal. Our federated ArcGIS Server environment seems to return results after it takes some time to validate against Portal. The current setup is:

OS - Windows Server 2016

HA web adaptors (load balancer in front; IWA enabled for portal) 

HA Portal 10.5.1 (no patches at the moment)

HA Mapping Server Federated

HA Host Server (federated)

We have about 2500 users within the environment. I found a blog post that sounded similar to my issue:

https://community.esri.com/thread/224966-portal-for-arcgis-1061-slow-after-upgrade-and-troubleshooti...

 

Unfortunately, this did not resolve the issue. I haven't seen anything in the logs, but I'm going to change the log level this evening when users are out to try and find more information. Has anyone every come across a similar issue? We will be removing the secondary machine from the site and seeing if the performance improves. We have observed anything out of the ordinary in the event logs either.

0 Kudos
1 Solution

Accepted Solutions
SamJehle1
New Contributor III

Looks as though the software FireEye (the xagt.exe) ended up being the culprit, but not exactly sure why at this point in time. Once the software was uninstalled, the issue was remedied. 

Should hopefully know more specifics in the near future.

View solution in original post

5 Replies
SamJehle1
New Contributor III

The issue is still on going, however we have noticed that the primary portal has a PostgreSQL Server instance that is constantly in use. The portal logs (debug) are not returning anything out of the ordinary:

The particular instance matches the PID found within the postmaster.pid file within the "..arcgisportal/db" folder. After failing over to the other portal server, the same issue occurs.

0 Kudos
SamJehle1
New Contributor III

Looks as though the software FireEye (the xagt.exe) ended up being the culprit, but not exactly sure why at this point in time. Once the software was uninstalled, the issue was remedied. 

Should hopefully know more specifics in the near future.

MichaelVolz
Esteemed Contributor

So is FireEye security software that your org installed to prevent malicious attacks on your servers (There have been many of those in the news over the past 6 months)?  Does this workaround now make your servers more vulnerable to attacks?

0 Kudos
MartinWennerwald
New Contributor

Did you ever learn what caused xagt.exe to result in the bad performance?

0 Kudos
SamJehle1
New Contributor III

FireEye was interrogating/monitoring portal's database which was causing the slow down. Once exceptions were made to exclude portal's database (typically C:\arcgisportal) FireEye was no longer a problem. We started treating FireEye the same was as McAfee and following ESRI recommended exception list:

FAQ: Which ArcGIS Enterprise directories should be excluded for security or antivirus software? 

0 Kudos