Latest Contributions by pfoppe

Hi @HarroldSompotan , Thanks for asking. On this specific system it was on a UNC share. This was one of our test systems. One additional factor, I have a CNAME record applied to the windows host and the config store uses the CNAME alias. That was done to allow us to move this around without having to try and remap everything through the Esri product. EX - File share is hosted on \\servername\sharename CNAME Alias applied and referenced as \\cname_alias\sharename For most of our mulit-machine deployments, they actually are mapped into DFS... many of these were created back at 10.1 and 10.2 when DFS was not prohibited and with reviews from Esri staff at conferences (like dev summit) the DFS concern never came up. In fact, there was positive feedback on the benefits afforded to us with DFS, primarily for moving things around on the back-end without having to move through the Esri product itself. Looking at the doc link you provided, I see the DFS is no longer supported starting at the 10.7 release. We have recently upgraded all our sites from 10.6.1 to 10.8.1 and missed this new requirement (couldn't find it in the whats new). This is concerning or us given our large ArcGIS server infrastructure is tied into DFS... so expect quite a bit of work in front of us to accommodate this (new to us) requirement. *sigh* ... *double sigh* -- edited -- I should have also mentioned... We are using both DFS and aliases (at times) along with very short share names due to path length issues we've run into in the past. If a publisher has a long service name, that ends up in the directories path mulitple times... so shorter the better...

Posting this to the community in hopes someone finds it useful if in the same situation we found ourselves. Problem Statement - ArcGIS Server v10.7.1 automatically disconnected from a one machine site due to configuration-store communication issues. Problem Details - ArcGIS Server running v10.7.1 in a single machine site. Config-store/directories were on a shared file system in preparation for additional machines to be added at a later date/time. Server was federated to 10.7.1 Portal through a web-adaptor hosted on IIS and designated as hosting server. Portal data store runs on a stand alone machine. Observations indicate there was some 'aggressive' security patching that caused multiple restarts of both the ArcGIS Server machine and the underlying file system where the configuration store and directories are hosted. Server logs indicated communication failures during the restart window, and ultimately disconnected from the site: <Msg time="2021-01-06T03:15:59,844" type="WARNING" code="7725" source="Server" process="1632" thread="1" methodName="" machine="<MACHINE><DOMAIN>" user="" elapsed="" requestID="">Verify machine registration observer: Disconnecting from site.</Msg> <Msg time="2021-01-06T03:15:59,844" type="WARNING" code="7725" source="Server" process="1632" thread="1" methodName="" machine="<MACHINE><DOMAIN>" user="" elapsed="" requestID="">Verify machine registration observer: Unable to verify current machine information after retries.</Msg> <Msg time="2021-01-06T03:15:54,844" type="WARNING" code="7725" source="Server" process="1632" thread="1" methodName="" machine="<MACHINE><DOMAIN>" user="" elapsed="" requestID="">Verify machine registration observer: Unable to get machine '<MACHINE><DOMAIN>' information from configuration store, retrying.</Msg> <Msg time="2021-01-06T03:15:49,797" type="WARNING" code="7725" source="Server" process="1632" thread="1" methodName="" machine="<MACHINE><DOMAIN>" user="" elapsed="" requestID="">Verify machine registration observer: Unable to get machine '<MACHINE><DOMAIN>' information from configuration store, retrying.</Msg> <Msg time="2021-01-06T03:15:44,766" type="WARNING" code="7725" source="Server" process="1632" thread="1" methodName="" machine="<MACHINE><DOMAIN>" user="" elapsed="" requestID="">Verify machine registration observer: Unable to get machine '<MACHINE><DOMAIN>' information from configuration store, retrying.</Msg> <Msg time="2021-01-06T03:15:39,719" type="WARNING" code="7725" source="Server" process="1632" thread="1" methodName="" machine="<MACHINE><DOMAIN>" user="" elapsed="" requestID="">Verify machine registration observer: Unable to get machine '<MACHINE><DOMAIN>' information from configuration store, retrying.</Msg> <Msg time="2021-01-06T03:15:34,672" type="WARNING" code="7725" source="Server" process="1632" thread="1" methodName="" machine="<MACHINE><DOMAIN>" user="" elapsed="" requestID="">Verify machine registration observer: Unable to get machine '<MACHINE><DOMAIN>' information from configuration store, retrying.</Msg> <Msg time="2021-01-06T03:15:29,625" type="WARNING" code="7725" source="Server" process="1632" thread="1" methodName="" machine="<MACHINE><DOMAIN>" user="" elapsed="" requestID="">Verify machine registration observer: Unable to get machine '<MACHINE><DOMAIN>' information from configuration store, retrying.</Msg> <Msg time="2021-01-06T03:15:29,406" type="INFO" code="7720" source="Server" process="1632" thread="1" methodName="" machine="<MACHINE><DOMAIN>" user="" elapsed="" requestID="">The cloud regions configuration file was deleted from this server machine. If you have other server machines in the site, please make sure that the file has been deleted from them as well.</Msg> <Msg time="2021-01-06T03:15:28,797" type="SEVERE" code="6599" source="Admin" process="1632" thread="1" methodName="" machine="<MACHINE><DOMAIN>" user="" elapsed="" requestID="">Failed to get the configuration of the server machine '<MACHINE><DOMAIN>'. Server machine '<MACHINE><DOMAIN>' is not registered with the Site.</Msg> <Msg time="2021-01-06T03:12:40,695" type="VERBOSE" code="7714" source="Admin" process="6076" thread="1" methodName="" machine="<MACHINE><DOMAIN>" user="" elapsed="" requestID="">Acquired 'machine-<MACHINE><DOMAIN>' workflow lock.</Msg> <Msg time="2021-01-06T03:12:40,617" type="WARNING" code="7704" source="Server" process="6076" thread="1" methodName="" machine="<MACHINE><DOMAIN>" user="" elapsed="" requestID="">Failed to stop web server. Cannot run program "cmd.exe" (in directory "C:\Program Files\ArcGIS\Server\framework\runtime\tomcat\bin"): CreateProcess error=19, The media is write protected</Msg> We normally have these file systems excluded from automatic patching to prevent issues (like this) but the exclusion list was inadvertently ignored. We normally patch these file systems manually and shut down ArcGIS Servers before the file system reboots. The following issues were also discovered - 1) File Missing - \\<machine>\C$\Program Files\ArcGIS\Server\framework\etc\config-store-connection.xml 2) File Inaccurate - \\<machine>\C$\Program Files\ArcGIS\Server\framework\etc\machine-config.xml Missing: <HTTPS>6443</HTTPS> 3) File Empty - \\<fileserver>\share\config-store\machines\<MACHINE>.<DOMAIN>.json Resolution: Ideally we would have a second machine in the site and simply 'join' the site, however, we had an underlying configuration-store on a shared path with no machines actively participating in it. Thought about trying to 'create a new site' using a local path, then manually overlay the existing config store (on the DFS) and try to 'move' it back. Alternatively a webgisdr restore may have resolved it. Instead, we found the following corrected the situation for us. 1) Shutdown ArcGIS Server 2) Copied a "config-store-connection.xml" file from a separate deployment into the installation path noted above. Updated file to reference the correct configuration store location (connectionString) 3) Manually edited the 'machine-config.xml' file to include the port 6443 4) Copied in a new <MACHINE>.<DOMAIN>.json file from a separate deployment into the configuration store path noted. Updated file to reference the correct machine (machineName & adminURL). Alternatively, we possibly could have extracted this from the last successful webgisdr backup file. 5) Start up Server The server did start up correctly, web-adaptor stayed connected, and was still federated to the portal and designated as the hosting server. All server validation checks passed and existing services were still functional. Hope this information helps if anyone runs in a similar/same issue.

‎12-09-2020

I did have an opportunity to meet with some Esri colleagues and figured I would sum up the sentiment here - The workflow is viable as long as we follow the vendor limitations. EX - only mapping/feature services with things like KML/WMS and even SOE/SOI are supported. No Image services, GP services, geodata services, etc We should expect to receive vendor support if we ran into issues given that the product manager (@pheede-esri) and product engineer (@Scott_M_MacDonald) crafted the referenced BLOG article and github hosted "sharedinstances.py" script There is still value in mass converting (or manually rebuilding) the ArcMap (.mxd) documents used to publish the services into an ArcGIS PRO project (.aprx). We are actively testing this out in our environment and will try to post back with experiences if I remember/have time In addition to those, we are still trying to fully understand the shared instance model and implications for management on our systems. Key areas we know some about but still trying to fully understand: "Caching Service Configurations in Memory" - How much 'memory' is used to 'cache' the service configurations? How much is too much, and what is a sustainable value (defaults to 50) Does the service get cached in all instances in the shared pool, or just the (1) that received the first request? How long does this take to 'cache' the service in memory? Note - we've done some initial testing and consistently saw an additional 1 second added to the first request response time.. subsequent requests ran faster. "Service Candidates" - What is the criteria to leverage shared instance? Or... alternatively, what criteria should indicate we should move back to 'dedicated' instances? "Less than XX hits per YY time"? Rational: If the service is frequently accessed, move to dedicated instances "Average response time less than AA per YY time"? Rational: If the service API calls are 'complex' (ex - lots of data, slow response times, etc) the run dedicated so it does not impact other services in the shared pool How do we track service usage and response times? Most likely through our web-server (ex - IIS) access logs correlating to the services in the shared instance pool. Then correlating to 'wait time' below "Wait Time" - We have observed 'wait time' metrics (ArcGIS Server "fine" level logging) to be reported on the 'dynamic mapping host' and NOT on the actual services themselves (for those services running in the shared instance pool). I suspect that wait-time would be experienced on ALL services in the shared instance pool if under load I think 'wait timeouts' were reported on the services in the shared instance pool, not the dynamic mapping host (needs to be confirmed still) Hope some of this helps. Thanks!

‎12-01-2020

Bump. Any thoughts on this? Thanks!

‎11-24-2020

Hello, We have a 5 machine ArcGIS Server v10.6.1 site hosting ~200 services and relying on "clustering" to manage machine VM size and isolate services from each other. Clusters have been deprecated at v10.7.x and we are assessing upgrade options (to v10.8.1). Each server runs in its own cluster and ~1/5 of the services are in each cluster (so each machine is hosting ~40 services). ~20-25% of the services are 'active' daily, while the other ~75-80% of the services have sporadic use. We believe the Esri "Shared Instance Pool" configuration is a candidate for the majority of the services on the stack. Problem is, this environment was built in JULY 2010 (literally within 30 days of ArcGIS Server v10.1 general availability) and ALL services published were from ArcMap. So... per the docs, it seems we would have to re-publish all services using ArcGIS PRO to take advantage of the new "Shared Instances" model. We have been looking into developing scripts to mass convert ArcMap documents (MXD) to ArcPRO projects (APRX), then re-publish with an 'overwrite' option so that we can take advantage of the shared instances. As part of this effort, we discovered that we can just make Admin API calls to 'changeProvider'. More specifically, it appears that an "ArcMap" published service sets the provider to "ArcObjects" while an ArcGIS PRO published service sets the provider to "ArcObjects11". A service running in the shared instance pool has a provider=DMaps. We were unable to directly update the provider from ArcObjects->DMaps (and vise versa DMaps -> ArcObjects), however... we were successful in making an API call to 1) convert provider=ArcObjects -> ArcObjects11 2) then a second API call to convert ArcObjects11 -> DMaps. As part of that, we did set the min/max instances to 0 (since thats what the 'manager' GUI did as well when manually converting a "PRO Published" service to the shared instance pool. We completed basic business testing (map export, query, identify, dynamic layers, etc) and have found no adverse impact yet. We are able to switch ArcMap->Pro->DMaps (and back in same direction) and can see the new ArcSOC.exe "command" path on the server change accordingly. So.... how viable of an approach is this? I suspect Esri support would indicate this is not a supported workflow (as my research has yet to find this) but so far seems completely functional. This conversion would allow us to more quickly support a v10.8.1 upgrade, although there is still value in getting our ArcMap documents converted to PRO for future publishing support (if we have to make changes to existing services and re-publish). Thanks for any info/advise you may have. Good Resources - Esri API Doc - "Change Provider" - https://developers.arcgis.com/rest/enterprise-administration/server/change-provider.htm Esri Doc - Configure Instance Settings - https://enterprise.arcgis.com/en/server/latest/administer/linux/configure-service-instance-settings.htm Esri BLOG - Move services to shared instances using python - https://www.esri.com/arcgis-blog/products/arcgis-enterprise/administration/move-services-to-shared-instances-using-python/ Esri GeoNet Post - Switch from Dedicated to Shared - https://community.esri.com/t5/arcgis-enterprise-questions/how-to-switch-all-services-from-dedicated-to-shared-in-one-go/m-p/240333

‎09-25-2020

Sounds like you are running "DBO Schema" database where your "dataowner" SQL account is either mapped as the actual database owner or added to the sysadmin role - A comparison of geodatabase owners in SQL Server—ArcMap | Documentation If you are publishing to server with this same SQL Account then yes, it sounds like your dataowner ("DBO") and account you use for publishing are one in the same. Double check the permission applied in your database against their documentation - Privileges for geodatabases in SQL Server—ArcMap | Documentation We missed a couple of these, and really discovered that a 'data editor' that is running the offline sync workflow actually needs the data creator permissions. specifically: CREATE TABLE CREATE PROCEDURE CREATE VIEW Users who create data must have a default schema with the same name as their database user name

‎09-24-2020

That is correct. For many of our published ArcGIS Server services - we use the "Operating System Authentication" (OSA) login when publishing. This is the Active Directory (AD) account that the ArcGIS Server runs as. Our data admins/publishers will grant the ArcGIS Server AD account the read/write permissions to the database objects. Most of these workflows are for read-only access - publish the service and let customers consume it for mashing up with other datasets. Although for some workflows (like the Esri offline 'feature server sync' capabilities) we use a SQL Server built-in account; we do NOT use the OSA account. The main driver for this was when the user syncs an offline map, there is a version created and owned by the account that was used to publish to server with (so the account the server uses to connect to SQL). If we used the OSA account, then it was challenging to manage that version without having full geodatabase administration ("sde" account) access. With the 'headless built-in SQL account', we can provide that account connection file to certain staff members who can then manage their back-end version by rec/posting to the parent version. The only way to do that with OSA would be to login to a computer with that same OSA account, or over-ride the version permissions to "PUBLIC", or give the user full SDE level admin access. So for the situation highlighted above... GIS Specialists use a built-in SQL account to 'own' the data. This is the account they would connect as when creating a feature class, granting permissions, applying indexes, making schema changes, etc. Lets call this account "dataowner". Objects in the database (lets call the database "mydatabase") end up looking like: mydatabase.dataowner.table_name The GIS specialists then use a different built-in SQL account to 'publish' the data to ArcGIS Server. This is the account they would connect with when publishing from ArcMap/ArcGIS PRO. Lets call this account "dataeditor". The GIS Spec will use the "dataowner" account to grant read/modify permissions on the "table_name" to the dataeditor account. When the user synchronizes the offline edit, then a version is usually created in the geodatabase that is owned by the 'dataeditor' account. EX - dataeditor.my_version The problem we ran into was that when the 'dataeditor' SQL account was created, it was inadvertently mapped to the schema owned by "dataowner". HTH.

‎09-23-2020

I realize this is an old thread and most likely not helpful for you, but posting for the community what we ran into.. We have a server side utility that uses the feature server 'apply edits' operations and that utility crafts a JSON payload based on the Esri API Spec. We ran into a similar response at v10.8.1 but there was an extended error code - https://server:443/arcgis/rest/services/folder/service/FeatureServer/6/applyEdits?f=json, STATUS_CODE 200, 47 ms, POST Buffer="{"addResults":[],"updateResults":[{"objectId":15468,"success":false,"error":{"code":1011,"extendedCode":-2147467259,"description":"Object is Missing."}}],"deleteResults":[]}" In this case, the server side utility had a bug where it was posting to the wrong layer ID (layer ID 6 in this case... it should have been layer id 0). I think the error message "Object is Missing" is really saying that the underlying "Object ID" in that layer is missing... Just some feedback. REF Feature service error codes—ArcGIS REST API | ArcGIS for Developers HTH

‎09-11-2020

We found another inconsistent configuration on one of our SQL Server environments. Posting here for the community. The account used to publish had the "Connect" permissions granted, but did NOT have the "Create Procedure", "Create Table", or "Create Function". See (redacted) screenshot: Find this in SQL Server Management Studio under "Databases" -> Right-Click the database -> "Properties" -> "Permissions" and choose the account that was published to the server

‎07-21-2020

I realize this thread is related to Oracle databases, but we had a similar error message with a back-end SQL Server environment. Cross posting for good measure... here was our fix on SQL Server - https://community.esri.com/thread/248923-collector-offline-map-sync-failed-unknown-error-occurred#comment-942437

Online Status	Offline
Date Last Visited	Thursday

My Ideas

Latest Contributions by pfoppe

Re: KB - ArcGIS Server auto removed from site due to Config-Store Communication Issues - FIXED

KB - ArcGIS Server auto removed from site due to Config-Store Communication Issues - FIXED

Re: Convert ArcMap published services (provider=ArcObjects) to PRO published services (provider=ArcObjects11) without re-publishing?

Re: Convert ArcMap published services (provider=ArcObjects) to PRO published services (provider=ArcObjects11) without re-publishing?

Convert ArcMap published services (provider=ArcObjects) to PRO published services (provider=ArcObjects11) without re-publishing?

Re: Collector Offline Map Sync Failed "Unknown error occurred"

Re: Collector Offline Map Sync Failed "Unknown error occurred"

Re: updateFeatures fails on Feature Layer

Re: Collector Offline Map Sync Failed "Unknown error occurred"

Re: ArcGIS Enterprise 10.6.1 Sync Service: No path for delta data changes