pfoppe

Intermittent slow performance: accessing rest page

Discussion created by pfoppe on Jul 9, 2012
Latest reply on Mar 11, 2013 by Justin_Rodriguez-esristaff
Problem: We have experienced intermittent slow performance issues when accessing the ArcGIS Rest endpoint.  Seems to happen when the GIS 'Site' has been unused for an extended amount of time (every morning - the first person to hit it), or if there have been configuration changes to the services (adding new services, changing existing services, changing permissions on services, etc). 

Below is a snippet from our IIS logs that shows the first request to the machine (notice 55 sec to 109 sec for 'time-take')

#Software: Microsoft Internet Information Services 7.5
#Version: 1.0
#Date: 2012-07-06 15:34:26
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken
2012-07-06 15:34:26 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 - YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 401 2 5 5375
2012-07-06 15:34:52 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 - YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 401 2 5 0
2012-07-06 15:36:33 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 DOMAIN\USERNAME YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 100812
...
...
...
2012-07-06 15:47:19 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 - YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1;+Trident/4.0;+.NET+CLR+1.1.4322;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.04506.30;+.NET+CLR+3.0.04506.648;+.NET+CLR+3.0.4506.2152;+.NET+CLR+3.5.30729;+InfoPath.3) 401 2 5 0
2012-07-06 15:48:14 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 DOMAIN\USERNAME YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1;+Trident/4.0;+.NET+CLR+1.1.4322;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.04506.30;+.NET+CLR+3.0.04506.648;+.NET+CLR+3.0.4506.2152;+.NET+CLR+3.5.30729;+InfoPath.3) 200 0 0 55187
...
...
...
#Software: Microsoft Internet Information Services 7.5
#Version: 1.0
#Date: 2012-07-09 14:25:16
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken
2012-07-09 14:26:08 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 DOMAIN\USERNAME YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 5173
2012-07-09 14:26:08 xxx.xxx.xxx.xxx GET /arcgis/rest/static/main.css - 80 DOMAIN\USERNAME YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 281
2012-07-09 14:27:35 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 DOMAIN\USERNAME YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 109718


I've enabled 'INFO' logging for the GIS Site and here is the only information from the same time period this morning:
Level Time Message Source Machine Code Time Elapsed User Name Process Thread Method Name 
INFO Jul 9, 2012 8:27:26 AM Request user: USERNAME, Service: null/null Rest HOSTNAME.DOMAIN 9029   796 1  
INFO Jul 9, 2012 8:25:46 AM Request user: USERNAME, Service: null/null Rest HOSTNAME.DOMAIN 9029   2296 1  
INFO Jul 9, 2012 8:25:18 AM Request user: USERNAME, Service: null/null Rest HOSTNAME.DOMAIN 9029   796 1  


Information about our configuration (high-availability):
    2 'web-serviers' running WIN 2008 R2 STD 64-bit with MS IIS7 and the ESRI ArcGIS Server Web Adaptor 10.1 (configured the same in a MS Network Load Balancer config)
    4 'GIS Servers' that are part of an ArcGIS 'Site'.  Each server is identical: WIN 2008 R2 STD 64-bit with ArcGIS Server 10.1 and ArcGIS Desktop 10.1
    Site has 3 clusters: 'Default', 'Mapping', 'GeoProcessing'
    The 'Mapping' and 'GeoProcessing' cluster has 2 machines in it.  Default is empty and no services are published to default.  Cannot remove the 'default' cluster. 
    the Site has user/role store configured to be maintained in the Windows Domain (AD)
    The authentication is at the 'web' tier
    The web-adaptor is configured on both 'web-servers' with the shared key from the site. 
    Not many services published right now.  Default install with 2 custom services (1 service has data registered w/ server, 1 service does not have data registered w/ server (it was copied to the server)

I've tested trying to access the site through both the MS NLB and accessing 1 specific node in the NLB (to rule out any LB issues).  I cannot access the rest-endpoint of the site via a server in the 'site' since we have configured security at the site.  We recieve the following message if trying to access a GIS site outside of the web-adaptor: http://localhost:6080/arcgis/rest/services
Error: WEB_ADAPTOR Authentication, User Credential is not set by the web adaptor for this request, check if the web adaptor is secured
Code: 403


We can publish services to the site by creating a .sd file and using the web manager to build/configure the service.  Services are accessible in products like ArcMap and ArcGIS Online, and other js viewers. 

Could it have to do with re-building the rest cache?  I've noticed that ArcGIS clears it automatically (when it needs to): http://resources.arcgis.com/en/help/main/10.1/#/What_s_new_in_ArcGIS_10_1_for_Server/015400000310000000/


Services Directory REST cache cleared automatically
Information in the ArcGIS Server Services Directory is cached so that you can have fast performance when navigating its links. In ArcGIS 10.1 for Server, if you add, remove, or modify a service, the REST cache is cleared automatically.


It does look like the rest-cache can still be cleared manually if needed though: http://resources.arcgis.com/en/help/rest/apiref/clearcache.html

Any help is appreciated!  Thanks!

Outcomes