Community

Intermittent slow performance: accessing rest page

5152
7
07-09-2012 07:50 AM
PF1
by
Occasional Contributor II
‎07-09-2012 07:50 AM
Problem: We have experienced intermittent slow performance issues when accessing the ArcGIS Rest endpoint.  Seems to happen when the GIS 'Site' has been unused for an extended amount of time (every morning - the first person to hit it), or if there have been configuration changes to the services (adding new services, changing existing services, changing permissions on services, etc). 

Below is a snippet from our IIS logs that shows the first request to the machine (notice 55 sec to 109 sec for 'time-take')

#Software: Microsoft Internet Information Services 7.5
#Version: 1.0
#Date: 2012-07-06 15:34:26
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken
2012-07-06 15:34:26 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 - YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 401 2 5 5375
2012-07-06 15:34:52 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 - YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 401 2 5 0
2012-07-06 15:36:33 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 DOMAIN\USERNAME YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 100812
...
...
...
2012-07-06 15:47:19 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 - YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1;+Trident/4.0;+.NET+CLR+1.1.4322;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.04506.30;+.NET+CLR+3.0.04506.648;+.NET+CLR+3.0.4506.2152;+.NET+CLR+3.5.30729;+InfoPath.3) 401 2 5 0
2012-07-06 15:48:14 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 DOMAIN\USERNAME YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1;+Trident/4.0;+.NET+CLR+1.1.4322;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.04506.30;+.NET+CLR+3.0.04506.648;+.NET+CLR+3.0.4506.2152;+.NET+CLR+3.5.30729;+InfoPath.3) 200 0 0 55187
...
...
...
#Software: Microsoft Internet Information Services 7.5
#Version: 1.0
#Date: 2012-07-09 14:25:16
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken
2012-07-09 14:26:08 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 DOMAIN\USERNAME YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 5173
2012-07-09 14:26:08 xxx.xxx.xxx.xxx GET /arcgis/rest/static/main.css - 80 DOMAIN\USERNAME YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 281
2012-07-09 14:27:35 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 DOMAIN\USERNAME YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 109718


I've enabled 'INFO' logging for the GIS Site and here is the only information from the same time period this morning: 
Level Time Message Source Machine Code Time Elapsed User Name Process Thread Method Name 
INFO Jul 9, 2012 8:27:26 AM Request user: USERNAME, Service: null/null Rest HOSTNAME.DOMAIN 9029   796 1  
INFO Jul 9, 2012 8:25:46 AM Request user: USERNAME, Service: null/null Rest HOSTNAME.DOMAIN 9029   2296 1  
INFO Jul 9, 2012 8:25:18 AM Request user: USERNAME, Service: null/null Rest HOSTNAME.DOMAIN 9029   796 1 


Information about our configuration (high-availability):
    2 'web-serviers' running WIN 2008 R2 STD 64-bit with MS IIS7 and the ESRI ArcGIS Server Web Adaptor 10.1 (configured the same in a MS Network Load Balancer config)
    4 'GIS Servers' that are part of an ArcGIS 'Site'.  Each server is identical: WIN 2008 R2 STD 64-bit with ArcGIS Server 10.1 and ArcGIS Desktop 10.1
    Site has 3 clusters: 'Default', 'Mapping', 'GeoProcessing'
    The 'Mapping' and 'GeoProcessing' cluster has 2 machines in it.  Default is empty and no services are published to default.  Cannot remove the 'default' cluster. 
    the Site has user/role store configured to be maintained in the Windows Domain (AD)
    The authentication is at the 'web' tier
    The web-adaptor is configured on both 'web-servers' with the shared key from the site. 
    Not many services published right now.  Default install with 2 custom services (1 service has data registered w/ server, 1 service does not have data registered w/ server (it was copied to the server)

I've tested trying to access the site through both the MS NLB and accessing 1 specific node in the NLB (to rule out any LB issues).  I cannot access the rest-endpoint of the site via a server in the 'site' since we have configured security at the site.  We recieve the following message if trying to access a GIS site outside of the web-adaptor: http://localhost:6080/arcgis/rest/services
Error: WEB_ADAPTOR Authentication, User Credential is not set by the web adaptor for this request, check if the web adaptor is secured
Code: 403


We can publish services to the site by creating a .sd file and using the web manager to build/configure the service.  Services are accessible in products like ArcMap and ArcGIS Online, and other js viewers. 

Could it have to do with re-building the rest cache?  I've noticed that ArcGIS clears it automatically (when it needs to): http://resources.arcgis.com/en/help/main/10.1/#/What_s_new_in_ArcGIS_10_1_for_Server/015400000310000...


Services Directory REST cache cleared automatically
Information in the ArcGIS Server Services Directory is cached so that you can have fast performance when navigating its links. In ArcGIS 10.1 for Server, if you add, remove, or modify a service, the REST cache is cleared automatically.


It does look like the rest-cache can still be cleared manually if needed though: http://resources.arcgis.com/en/help/rest/apiref/clearcache.html

Any help is appreciated!  Thanks!
Tags (2)
7 Replies
PF1
by
Occasional Contributor II
‎07-10-2012 11:05 AM
Update: After spending 2 full days trying to pin down our performance issues... We found some ways to replicate the issues, but are still working on a solid resolution..


Configuration 1 - Users/roles:Win. domain & Authentication:Web Tier = Horrible Performance
We had the site configured like this: http://forums.arcgis.com/threads/61507-Problem-with-10.1-Web-Adaptor#6

Basically security was enabled to have the user/role store in a 'windows domain' and the security authentication was at the 'web tier'. You can see with the IIS logs that performance was horribly bad when accessing the rest home page at http://<web-adaptor_HOSTNAME>/arcgis/rest/services ... it would take 60-120 seconds to render this page and return the directory listing. It would also prompt for a username/password for the first person who accessed the site (and error with a HTTP 401 response - see line 1 from the IIS log below), but then succeed (after 60-120 sec) on subsequent requests (see line 2 in IIS log):

#Software: Microsoft Internet Information Services 7.5
#Version: 1.0
#Date: 2012-07-10 17:33:46
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken
2012-07-10 17:33:46 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 - YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 401 1 2148074254 375
2012-07-10 17:35:25 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 DOMAIN\USERNAME YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 78140


Configuration 2 - Users/roles:Win. domain & Authentication:GIS Tier (Providers... 'Negotiate' at the top) = Horrible Performance
re-configuring the security to use 'windows domain' for the user/role store, but having the authentication at the 'GIS SERVER'. We left the 'arcgis' virtual directory in IIS configured to disable anonymous access and enable windows authentication (with NTLM below Negotiate in the 'providers' section) and still saw horrid performance. 
#Software: Microsoft Internet Information Services 7.5
#Version: 1.0
#Date: 2012-07-10 17:46:10
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken
2012-07-10 17:50:34 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 - YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 203656


Configuration 3 - Users/roles:Win. domain & Authentication:GIS Tier (Providers... 'NTLM' at the top) = Horrible Performance
Same as above (AD for users/roles, but GIS server for authentication) except that we moved NTLM above Negotiate in the 'providers' section and this seems to still have horrible performance: 
#Software: Microsoft Internet Information Services 7.5
#Version: 1.0
#Date: 2012-07-10 17:52:51
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken
2012-07-10 17:57:03 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 - YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 99750
2012-07-10 18:01:14 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 - YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 92578


Configuration 4 - Users/roles:GIS Server & Authentication:GIS Tier (DEFAULT INSTALL) = GREAT PERFORMANCE
About the only way to acheive adaquate performance is to use the defaults out of the box (GIS Server for user/role store and for authentication). The problem is that we would really like to tie in our AD credentials for administration/publishing and be able to track use by users in our orginization as described here: Securing your ArcGIS Server site


To get back to defults: Login to IIS and set the 'arcgis' virtual directory to allow anonymous auth. and disable the 'Windows auth.'. Then go to the location of our 'config-store'->security and edit the 'security-config.json' file to look like this: 
{
  "securityEnabled": true,
  "authenticationMode": "ARCGIS_TOKEN",
  "authenticationTier": "GIS_SERVER",
  "userStoreConfig": {
    "type": "BUILTIN",
    "properties": {}
  },
  "roleStoreConfig": {
    "type": "BUILTIN",
    "properties": {}
  },
  "sslEnabled": false,
  "httpEnabled": true,
  "virtualDirsSecurityEnabled": false
}


and then log into arcgis/manager and setting the permissions at the root to be 'public, available to everyone'. Re-config the web-adaptor on the IIS server and the site flys!!!

#Software: Microsoft Internet Information Services 7.5
#Version: 1.0
#Date: 2012-07-10 18:42:57
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken
2012-07-10 18:42:57 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 - YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 734
2012-07-10 18:43:02 xxx.xxx.xxx.xxx GET /arcgis/rest/services/Basemaps_ags/MapServer - 80 - YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 2562
2012-07-10 18:43:11 xxx.xxx.xxx.xxx GET /arcgis/rest/services/Basemaps_ags/MapServer/export bbox=-158.07694190315246,48.12808111415401,-61.13182215488604,83.76600431210284 80 - YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 6187
2012-07-10 18:43:11 xxx.xxx.xxx.xxx GET /arcgis/rest/directories/arcgisoutput/Basemaps_ags_MapServer/_ags_map23a9496fa48b4d8583d9a0d53fec08cf.png - 80 - YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 31
2012-07-10 18:43:14 xxx.xxx.xxx.xxx GET /arcgis/rest/services - 80 - YYY.YYY.YYY.YYY Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.1;+WOW64;+Trident/4.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3) 200 0 0 328




Any support for getting acceptable performance using the 'web tier' for authentication would be much appreciated. We would rather have our IIS server handle the authentication, but for now will leave it with the GIS server handle the authentication.
0 Kudos
PF1
by
Occasional Contributor II
‎07-10-2012 12:14 PM
OK - last update for a while as I think we are going to try this configuration out.  Seems to be the best solution regarding stability and meeting our requirements:

From the standard install (what comes out of the box with no security configured):

  • We have enabled the user/role store to be 'windows domain'

  • We have left the 'GIS Server' for the authentication

  • IIS 'arcgis' virtual directory Settings: Anonymous is enabled & 'Windows Authentication' is disabled

  • Configured a 'administrator' AD group and a 'publisher' AD group. 


Performance seems very good.  Accessing the rest page comes up in usually less than 1 second. 

There are a few draw-backs which we can deal with for now:

  • Users are not automatically authenticated which means that we cannot track who is using what services (minor deal)

  • Publishers and Administrators are not automatically authenticated.  They need to store their credentials in the arcgis server connection file which could pose issues when their passwords expire (if they want to administer/publish from the ArcGIS Desktop products). 


Now just need to test this with configuring the second web-server (w/ web-adaptor) behind the NLB and see if it still performs well...
0 Kudos
TysonHaverkort
by
New Contributor III
‎10-04-2012 12:49 PM
Thanks a ton for posting this, and taking the time to post the resolution.  I had a similar issue today, except instead of poor performace, I was locked out of my REST services completely.  I change the Authentication Tier from IIS back to GIS Server, and it worked.  Same as you, not ideal, but it'll have to do.
0 Kudos
BrianLeroux
by
Occasional Contributor III
‎11-29-2012 09:58 AM
Patrick-
Have you had a chance to see if the slowness issues have been resolved with SP1?
0 Kudos
JustinRodriguez
by
Occasional Contributor
‎02-13-2013 08:01 AM
Hello,
Do you have a large amount of security groups in your AD? I am wondering if your header length could be the cause of the issue.
Thanks-
0 Kudos
KirkMower
by
New Contributor III
‎03-08-2013 04:46 PM
We are having the same issues. Apparently ArcGIS Server (10.1 with all service packs) needs a jump start in the morning. That is, it seems to have fallen asleep overnight as it goes idle. The problem isn't so much the the fact that it is sleeping, but that it doesn't wake up on the web calls from the web-based application. It seems to require a harder kick from 'Mom and Dad', that is us, by hitting the REST endpoint directly from the browser. This isn't consistent behaviour either, as it seems to have 'big nights out' where it doesn't wake up, and other mornings where it stayed at home studying...

I suspect that this is a 'feature' to get ArcGIS to run more 'politically correct' for those shortsighted IT managers that have been sold the dream of virtualization at all costs. I don't think those of us who have the foresight to buy a physical server and have a good solid three-tier architecture should have to pay for this. We have been down that road of explaining why virtualization just doesn't work well enough many times. So, can we please have a setting that allows us to keep the service running and alert at all times, even if it does take up extra CPU and memory, with the full knowledge that the CPU and memory is all ours anyway? I think that ESRI has cut server off at the knees to please performance hedging numpties in shared environments.
0 Kudos
JustinRodriguez
by
Occasional Contributor
‎03-11-2013 07:34 AM
We are having the same issues. Apparently ArcGIS Server (10.1 with all service packs) needs a jump start in the morning. That is, it seems to have fallen asleep overnight as it goes idle. The problem isn't so much the the fact that it is sleeping, but that it doesn't wake up on the web calls from the web-based application. It seems to require a harder kick from 'Mom and Dad', that is us, by hitting the REST endpoint directly from the browser. This isn't consistent behaviour either, as it seems to have 'big nights out' where it doesn't wake up, and other mornings where it stayed at home studying...

I suspect that this is a 'feature' to get ArcGIS to run more 'politically correct' for those shortsighted IT managers that have been sold the dream of virtualization at all costs. I don't think those of us who have the foresight to buy a physical server and have a good solid three-tier architecture should have to pay for this. We have been down that road of explaining why virtualization just doesn't work well enough many times. So, can we please have a setting that allows us to keep the service running and alert at all times, even if it does take up extra CPU and memory, with the full knowledge that the CPU and memory is all ours anyway? I think that ESRI has cut server off at the knees to please performance hedging numpties in shared environments.


Kurt,
Your issue might have similar symptoms, but I do not think the cause is related to the posts above. Your problem can most likely be solved. Please call (888)377-4575 and ask one of the Server Usage Analysts for assistance. They should be able to help you. Take care-
Justin
0 Kudos