Latest Contributions by PF1

We have an existing ArcGIS Server  (AGS) 10.0 solution that is hosting close to 1,000 mapping services.  We have been working on an upgrade to this environment to 10.2.1 for a few months now and we are having a hard time getting a stable environment.  These services have light use, and our program requirements are to have an environment that can handle large amounts of services with little use.  In the AGS 10.0 space we would set all services to 'low' isolation with 8 threads/instance.  We also had 90% of our services set to 0 min instances/node to save on memory.  Below is a summary of our approaches and where we are today.  I'm posting this to the community for information, and I am really interested in some feedback and or recommendations to make this move forward for our organization. Background on deployment: Targeted ArcGIS Server 10.2.1 Config-store/directories are hosted on a clustered file server (active/passive) and presented as a share: \\servername\share Web-tier authentication 1 web-adaptor with anonymous access 1 web-adaptor with authenticated access (Integrated Windows Authentication with Kerberos and/or NTLM providers) 1 web-adaptor 'internally' with authenticated access and administrative access enabled (use this for publishing) User-store: Windows Domain Role-store: Built-In We have a few arcgis server deployments that look just like this and are all running fairly stable and with decent performance.  Approach 1: Try to mirror (as close as possible) our 10.0 deployment methodology 1:1 Build 4 AGS 10.2.1 nodes (virtual machines).     Build 4 individual clusters & add 1 machine to each cluster Deploy 25% of the services to each cluster.  The AGS Nodes were initially spec'd with 4 CPU cores and 16GB of RAM. Each ArcSOC.exe seems to consume anywhere from 100-125MB of RAM (sometimes up to 150 or as low as 70).  Publishing 10% of the services with 1 min instance (and the other 90 to 0 min instances) would leaving around 25 ArcSOC.exe on each server when idle.  The 16GB of RAM could host a total of 100-125 total instances leaving some room for services to startup instances when needed and scale slightly when in use. our first problem we ran into was publishing services with 0 instances/node.  Esri confirmed 2 'bugs': #NIM100965 GLOCK files in arcgisserver\config-store\lock folder become frozen when stop/start a service from admin with 0 minimum instances and refreshing the wsdl site #NIM100306 : In ArcGIS Server 10.2.1, service with 'Minimum Instances' parameter set to 0 gets published with errors on a non-Default cluster So... that required us to publish all of our services with at least 1 min instance per node.  At 1,000 services that means we needed 100-125GB of ram for all the ArcSOC.exe processes running without any future room for growth.... Approach 2: Double the RAM on the AGS Nodes We added an additional 16GB of RAM to each AGS node (they now have 32GB of RAM) which should host 200-250 arcsoc.exe (which is tight to host all 1,000 services).  We published about half of the services (around 500) and started seeing some major stability issues.  During our publishing workflow... the clustered file server would crash.  This file server hosts the config-store/directories for about 4 different *PRODUCTION* arcgis server sites.  It also hosts our citrix users work spaces and about 13TB of raster data.  During a crash, it would fail-over to the passive file server and after about 5 minutes the secondary file server would crash.  This is considered a major outage! On the last crash, some of the config-store was corrupted.  While trying to login to the 'admin' or 'manager' end-points, we received an error that had some sort of parsing issue.  I cannot find the exact error message.  We had disabled the primary site admin account, so went in to re-enable, but the super.json file was EMPTY!  We had our backup team restore the entire config-store from the previous day, and copied over the file.  I'm not sure what else was corrupted.  after restoring that file we were able to login again with our AD accounts.  The file-server crash was clearly caused by publishing a large amounts of services to this new arcgis server environment.  We caused our clustered file servers to crash 3 separate times all during this publishing workflow.  We had no choice but to isolate this config-store/directories to an alternate location.  We moved it to a small web-server to see if we could simulate the crashes there and continue moving forward.  So far it has not crashed that server since.  During bootups, with the AGS node hosting all the services, the service startup time was consistently between 20 and 25 minutes.  We were able to find a start-up timeout setting at each service that was set to 300 seconds (5 minutes) by default.  we set that to 1800 seconds (30 minutes) to try and get these machines to start-up properly.  What was happening is that all the arcsoc.exe processes would build and build until some point they would all start disappearing.  In the meantime, we also reviewed the ArcGIS 10.2.2 Issues Addressed List which indicated: NIM099289 Performance degradation in ArcGIS Server when the location of the configuration store is set to a network shared location (UNC). We asked our Esri contacts for more information regarding this bug fix and basically got this: …our product lead did provide the following as to what updates we made to address the following areas of concern listed inNIM099289: 1.       The Services Directory 2.       Server Manger 3.       Publishing/restarting services 4.       Desktop 5.       Diagnostics ArcGIS Server was slow generating a list of services in multiple places in the software.  Before this change, ArcGIS Server would read from disk all services in a folder every time the a list of services was needed - this happened in the services directory, the manager, ArcCatalog, etc.  This is normally not that bad, but if you have many many services in a folder, and you have a high number of requests, and your UNC/network is not the fastest, then this can become very slow.  Instead we remember the services in a folder and only update our memory when they have changed. Approach 3: Upgrade to 10.2.2 and add 3 more servers We added 3 more servers to the 'site' (all 4CPU, 32GB RAM) and upgraded all to 10.2.2.  We actually re-built all the machines from scratch again We threw away our existing 'config-store' and directories since we knew at least 1 file was corrupt.  We essentially started from square 1 again.  All AGS nodes were installed with a fresh install of 10.2.2 (confirmed that refreshing folders from REST page were much faster).  Config-store still hosted on web-server We mapped our config-store to a DFS location so that we could move it around later Published all 1,000 ish services successfully with across 7 separate 'clusters' Changed all isolation back to 'high' for the time being.  This is the closest we have gotten.  At least all services are published.  Unfortunately it is not very stable.  We continually receive a lot of errors, here is a brief summary:      Level Message Source Code Process Thread SEVERE Instance of the service '<FOLDER>/<SERVICE>.MapServer' crashed. Please see if an error report was generated in 'C:\arcgisserver\logs\SERVERNAME.DOMAINNAME\errorreports'. To send an error report to Esri, compose an e-mail to ArcGISErrorReport@esri.com and attach the error report file. Server 8252 440 1 SEVERE The primary site administrator '<PSA NAME>' exceeded the maximum number of failed login attempts allowed by ArcGIS Server and has been locked out of the system. Admin 7123 3720 1 SEVERE ServiceCatalog failed to process request. AutomationException: 0xc00cee3a - Server 8259 3136 3373 SEVERE Error while processing catalog request. AutomationException: null Server 7802 3568 17 SEVERE Failed to return security configuration. Another administrative operation is currently accessing the store. Please try again later. Admin 6618 3812 56 SEVERE Failed to compute the privilege for the user 'f7h/12VDDd0QS2ZGGBFLFmTCK1pvuUP1ezvgfUMOPgY='. Another administrative operation is currently accessing the store. Please try again later. Admin 6617 3248 1 SEVERE Unable to instantiate class for xml schema type: CIMDEGeographicFeatureLayer <FOLDER>/<SERVICE>.MapServer 50000 49344 29764 SEVERE Invalid xml registry file: c:\program files\arcgis\server\bin\XmlSupport.dat <FOLDER>/<SERVICE>.MapServer 50001 49344 29764 SEVERE Unable to instantiate class for xml schema type: CIMGISProject <FOLDER>/<SERVICE>.MapServer 50000 49344 29764 SEVERE Invalid xml registry file: c:\program files\arcgis\server\bin\XmlSupport.dat <FOLDER>/<SERVICE>.MapServer 50001 49344 29764 SEVERE Unable to instantiate class for xml schema type: CIMDocumentInfo <FOLDER>/<SERVICE>.MapServer 50000 49344 29764 SEVERE Invalid xml registry file: c:\program files\arcgis\server\bin\XmlSupport.dat <FOLDER>/<SERVICE>.MapServer 50001 49344 29764 SEVERE Failed to initialize server object '<FOLDER>/<SERVICE>': 0x80043007: Server 8003 30832 17 Other observations: Each AGS node makes 1 connection (session) to the file-server containing the config-store/directories During idle times, only 35-55 files are actually open from that session. During bootups (and bulk administrative operations), the file's open jump consistently between 1,000 and 2,000 open files per session The 'system' process on the file server spikes especially during bulk administrative processes.  The AGS nodes are consistently in communication with the file server (even when the site is idle).  CPU/Memory and Network monitor on that looks like this: AGS nodes look similar.  It seems there is a lot of 'chatter' when sitting idle.  Requests to a service succeed 90% of the time but 10% of the time we receive HTTP 500 errors: Error: Error exporting map Code: 500 Options for the future We have an existing site with the ArcGIS SOM instance name of 'arcgis'.  These 1,000 services are running in that 10.0 site for the past few years.  Users have interacted with this using a URL like: http://www.example.com/arcgis/rest/services/<FOLDER>/<MapService>/MapServer We are trying to host all these same services so that users accessing this URL will be un-impacted.  If we cannot, we will switch to 1 server in 1 cluster in 1 site (and instead have 7 sites).  We will then be re-publishing all our content to individual sites but will have different URL's: http://www.example.com/arcgis1/rest/services/<FOLDER>/<MapService>/MapServer http://www.example.com/arcgis2/rest/services/<FOLDER>/<MapService>/MapServer ... ... http://www.example.com/arcgisN/rest/services/<FOLDER>/<MapService>/MapServer We would have extensive amount of work to either (or both) communicate all the new URL's to our end users (and update all metadata, products, documentation, and content management systems to point to the new URL's) and/or build URL Re-direct (or URL Re-write) rules for all the legacy services.  Neither of two options are ideal, but right now we seem to have exhausted all other options.  Hopefully this will help other users while they troubleshoot thier arcserver deployment.  Any ideas are greatly appreciated with our strategy to make this better.  Thanks! ... View more

We have public facing web-services with the 'feature access' capability enabled.  Operations include create, update, delete (and geometry updates).  We are also attempting various offline capabilities using the newly offered 'sync' operation.  We are unable to get any of this to work using web-tier authentication (NTLM, Kerberos,Http Basic, etc).  The public facing web-services are configured similar to the Multiple firewalls with reverse proxy and Web Adaptor in a perimeter network on the ArcGIS server help documentation: web-tier authentication User store: windows domain role store: built-in web-adaptor server sitting in our DMZ GIS Site sitting in our internal network Reverse proxy communication from DMZ to internal network.  Web-app Firewalls (WAF) in front of and behind the web-adaptor server in the perimeter DMZ environment on the web-adaptor server we have deployed two web adaptors to Supporting a mix of public and private services.  One web-adaptor is deployed over both port 80 and 443 but allows strictly anonymous access (for consuming and unauthenticated access).  The second web-adaptor we have anonymous access disabled and have enabled Integrated Windows Auth (IWA) using both kerberos and NTLM as providers.  We have also tested using HTTP Basic.  If we add feature service content to an arcgis online map, it tells us that its an internal only service and editing is disabled.  The services are accessible from the dirty internet.  It appears that arcgis.com map executes a request to the service info page (https://www.myserver.com/arcgisauth/rest/info?f=json) and also tries to proxy the request like so: Request URL:https://www.arcgis.com/sharing/proxy?https://www.myserver.com/arcgisauth/rest/services/FeatureServices/MyService/FeatureServer/0?f=json Request Method:POST Status Code:504 Gateway Timeout Both of those fail because the public facing server returns an HTTP Error code 401 with the 'www-authenticate' headers as the options the client has available to authentication.  We have tried Kerberos, NTLM, and HTTP Basic.  It appears that the arcgis.com map ignores he 'www-authenticate' header and just disables the editing capabilities rather than attempting to obtain the user credentials.  We can successfully configure the public facing web-server to use GIS Token based authentication (anonymous enabled and IWA disabled on the web-adaptor server), but that security configuration is really not ideal for our customer base.  Is this a known limitation?  Is there something we are doing wrong?  I would have expected web-tier authenticated services to have editable capabilities if users supplied their credentials. Thanks for any help/guidance! ... View more

Many organizations use the Microsoft Active Directory (AD) product for users identity store (name, info, password), to authenticate those users credetials (verify they provided the correct credentials) and then subsequently authorize those users to perform various actions (based on role or group membership).   The Esri ArcGIS Server 10.2 release started working with multiple domains inside 1 AD forest.   I can have users in different domains authenticate and be authorized to GIS resources with 1 deployment of the Esri ArcGIS Server 'site'.   Large organizations often times build 'forest trusts' between different Active Directory deployments.  This allows organizations to authenticate and subsequently authorize users from a different corporate identity stores access to resouces within the organizations control without having to establish an Identity Life Cycle Management process with those other identities.  Many 3rd party products that use 'Windows Domain' based authentication/authorization schemes support Active Directory forest tust models out of the box.  It appears that Esri ArcGIS Server completly igores the trust.  The web-tier (web-adaptor) can be configured to authenticate users across an AD trust (for example, using Integrated Windows Authentication).  The ArcGIS Server that handels the authorization (what service the user is allowed access to) does not know what to do with users that are in the trusted AD forest.  The only alternative I see is to build a custom identity provider that queries all domains within each forest, which is not really an out of the box solution.   Thanks for the consideration.   ... View more

I help to administer an ArcGIS Online Organizational solution.  We have close to 100 members or so from our enterprise that can login to that.  Our user accounts are NOT using the enterprise login feature (like SAML back to our internal AD structure), so we are just using built-in identity provider (Esri GLOBAL accounts?!).  The Esri ArcGIS Online product appears to have been vulnerable to the OpenSSL Vulnerability CVE-2014-0160 (Heartbleed) as documented on the Esri Knowledge Base and subsequently patched with new CA certs: �?� ArcGIS Online �?? Mitigations have been applied to all service endpoints and certificates have been re-issued across the platform. As a precautionary measure, Esri encourages users to change passwords for systems where mitigations have been completed, such as ArcGIS Online. Is there a way I can force the user to change their password on next login for all my users in the ORG solution?  I can blast out emails asking them to, but I have not yet found a way to force it.  If I cannot force them, is there a way I can get notified that they have changed their password for our tracking purposes?  I know I could disable the user, but cannot find a better way to mitigate the risks of user accounts being compromised.  It does appear that it might be possible to set the users password with a random one, provide that to the user, have them re-set it, and track progress that way.  might be possible through the update user operation as its listed in the user parameters section (although I have not tested this yet).  Thanks in advance for any guidance! ... View more

Just posting for FYI in case someone comes across the same problem... Running Esri ArcGIS Server 10.1 SP1 in a distributed environment (2 network load balanced web-adaptors and 4 GIS servers in a 2 cluster site) https://HOST:6443/arcgis/admin/?f=pjson {   "resources": [     "machines",     "clusters",     "system",     "services",     "security",     "data",     "uploads",     "logs"   ],   "currentVersion": 10.11,   "fullVersion": "10.1.1",   "acceptLanguage": "en-US,en;q=0.8" } The problem: I was working with the ArcGIS Server Administrative API to do some simple service configurations (specifically I was knocking down the min # of instances from 1 to 0) to free up some memory resources on our dev/test environment that has a lot of web-service bloat.  I was executing an HTTP Post that looked something like this: https://HOST:6443/arcgis/admin/services/FOLDER/SERVICE.MapServer/edit with a payload like this: {"minInstancesPerNode": 0} and the response from the server was: {"status":"error","messages":["Could not undeploy services from one or more machines. 'Server machine 'https://HOST.DOMAIN:6443/arcgis/admin' returned an error. 'C:\\Program Files\\ArcGIS\\Server\\framework\\runtime\\ejbs\\FOLDER\\SERVICE.MapServer\\Recycling (Access is denied)''."]} Tried the process multiple times.  Could not find anything of significant value in the Event Viewer on the server throwing the fit.  Couple of quick google searches resulted in someone having a similar problem and restarting the machine fixed it.  I dont want to restart the machine... so the witch-hunt started to find a less invasive solution. The Fix: I decided to try clearing the REST cache.  Clicked the button Clear Cache at the following URL: https://HOST:6443/arcgis/admin/system/handlers/rest/clear provided a good result: {"status": "success"} And presto... I could immediately adjust server settings again. Hope this little write-up saves someone time. UPDATE - It appears that my issue is still here...  It might just be one of the machines in my site that fails as the other machines are not throwing an error.  I only get the error on 1 machine in my site.  I will keep troubleshooting, but maybe rebooting just the 1 machine will work. ... View more