ARCGISUtility.psm1 failed error on ArcGIS Enterprise Deployment in Azure using Cloud Builder

[TCAPublicAccount]

Hi All,

I am trying to deploy ArcGIS Enterprise 10.8.1 base deployment in Azure using Cloud Builder 10.8.1 with the following parameters:

1. ArcGIS services account as a domain account

2. CA signed certificate 

3. Azure Files(SMB) as the Azure cloud storage for the configuration and content store.

4. Relational data Store

5. 4 VM's - 1 each for Portal, Server, Data Store and FileShare.

The deployment fails with the below error:

The resource taServer-0/DSCConfiguration0 of type Microsoft.Compute/virtualMachines/extensions failed.

 Error:- The resource operation completed with terminal provisioning state 'Failed'.

<b>VMExtensionProvisioningError</b> - VM has reported a failure when processing extension 'DSCConfiguration0'. Error message: "DSC Configuration 'ServerConfiguration' completed with error(s). Following are the first few: PowerShell DSC resource ArcGIS_Server  failed to execute Set-TargetResource functionality with error message: ERROR: ArcGISUtility.psm1 failed. Failed to return security configuration. null  The SendConfigurationApply function did not succeed."

Prior to this error message, the log files also has a record of the following Warning messages:

[02/10/2021 12:04:58] [VERBOSE] [taServer-0]:                            [[ArcGIS_Server]Server] Response from CreateSite:- Failed to create the site. Configuration store error. Cannot get persistence. File system '\\gisconfigstore.file.core.windows.net\fileshare\gisportal\server\config-store' connection failed. The specified location is not accessible. Ensure that the ArcGIS Server account has read and write access to the location.

[02/10/2021 12:04:58] [VERBOSE] [taServer-0]:                            [[ArcGIS_Server]Server] [WARNING] Error while creating site on attempt 1 Error:- CreateSite Failed. Error:- Failed to create the site. Configuration store error. Cannot get persistence. File system '\\gisconfigstore.file.core.windows.net\fileshare\gisportal\server\config-store' connection failed. The specified location is not accessible. Ensure that the ArcGIS Server account has read and write access to the location.

[02/10/2021 12:04:58] [VERBOSE] [taServer-0]:                            [[ArcGIS_Server]Server] [WARNING] Unable to create Site. Error:- CreateSite Failed. Error:- Failed to create the site. Configuration store error. Cannot get persistence. File system '\\gisconfigstore.file.core.windows.net\fileshare\tcagisportal\server\config-store' connection failed. The specified location is not accessible. Ensure that the ArcGIS Server account has read and write access to the location.

[02/10/2021 12:04:58] [VERBOSE] [taServer-0]:                            [[ArcGIS_Server]Server] CreateSite Failed. Error:- Failed to create the site. Configuration store error. Cannot get persistence. File system '\\tcagisconfigstore.file.core.windows.net\fileshare\tcagisportal\server\config-store' connection failed. The specified location is not accessible. Ensure that the ArcGIS Server account has read and write access to the location.

Has anybody faced a similar problem or know of a resolution to this? I have tried the deployment multiple times but each time its returning the same : "ArcGISutility.psm1 failed" error message. 

Looking forward to hear from the community for any insights or suggestions to go beyond this error message to get the deployment done . 

[Anonymous User]

Hi @TCAPublicAccount ,

Seems your provided ArcGIS Server account doesn't have read and write access permission to the file share (Fileserver). Check the security group permissions of your resources group.

Manoj

[TCAPublicAccount]

Hi @Anonymous User ,

Thanks for responding to my query . The file share is created on the fly while running Cloud Builder and as a part of the Cloud Builder execution. Do you mean ArcGIS Service account needs to be granted a role on the resource group, if yes, what role? Can you please elaborate on the statement when you say "Check the security group permissions on the resource group"?

Looking forward to hearing from you.

[DavidHoy]

just a stab - does the password you have provided for the "ArcGIS Service Account" meet Azure requirements (length, characters and numbers, special characters etc)?
Just possibly this is causing the ArcGIS Server account to not be created and that means the Azure configstore file share is not accessible for the AGS service

[TCAPublicAccount]

Hi @DavidHoy 

Thanks for responding to my query. The ArcGIS Service Account is already created as a domain account. The password of ArcGIS Service account adheres to Azure requirements. Its 16 characters in length with lower and upper case letters, numbers and allowed special characters.

The error occurs when the cloud builder script is trying to create a site and is trying to use File System based cloud storage for the config store. Apart from Resource Group, Vnet , subnet, application gateway subnet all other resources are created on the fly in Azure using Cloud Builder. 

[TCAPublicAccount]

UPDATE: 

I tried the deployment using the below two scenarios but the deployment is failing with the similar error which is quite intriguing to me. If anyone has any idea about it kindly let me know.

Scenario1 :

• Used domain join option with arcgis service account as a domain account and used Azure Cloud Storage for the configuration and content store.
• While Cloud Builder was running, once the logs reached to the point where all VM’s are joined to the Domain,

1)  I created a fileshare folder manually in Azure Portal in the Azure Cloud Storage used for Configuration and content store ( because in my earlier trials fileshare folder was not getting created and the deployment used to fail while trying to create a site while accessing Azure Files during Server configuration) ;

2)  logged in to Server, Portal, Data Store and FileShare VM’s with admin credentials and granted Arcgis user the admin privileges by adding Arcgis user in Administrator group.

Result :

With the above manual changes, the deployment passed the arcgis account access issue and the problem of creating config and content store folders in fileshare of Azure Cloud Storage account but it failed at Portal configuration with a weird error :

02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]: LCM:  [ Start  Test     ]  [[ArcGIS_WindowsService]ArcGISGeoEventGateway_Service]

[02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]:                            [[ArcGIS_WindowsService]ArcGISGeoEventGateway_Service] User name for service 'ArcGISGeoEventGateway' is 'LocalSystem'. It does not match 'tca\arcgis.

[02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]: LCM:  [ End    Test     ]  [[ArcGIS_WindowsService]ArcGISGeoEventGateway_Service]  in 0.0000 seconds.

[02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]: LCM:  [ Start  Set      ]  [[ArcGIS_WindowsService]ArcGISGeoEventGateway_Service]

[02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]:                            [[ArcGIS_WindowsService]ArcGISGeoEventGateway_Service] Service 'ArcGISGeoEventGateway' already stopped, no action required.

[02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]: LCM:  [ End    Set      ]  [[ArcGIS_WindowsService]ArcGISGeoEventGateway_Service]  in 0.0310 seconds.

[02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]: LCM:  [ End    Resource ]  [[ArcGIS_WindowsService]ArcGISGeoEventGateway_Service]

[02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]: LCM:  [ Start  Resource ]  [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service]

[02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]: LCM:  [ Start  Test     ]  [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service]

[02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]:                            [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service] User name for service 'ArcGIS Notebook Server' is 'LocalSystem'. It does not match 'tca\arcgis.

[02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]: LCM:  [ End    Test     ]  [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service]  in 0.0000 seconds.

[02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]: LCM:  [ Start  Set      ]  [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service]

[02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]:                            [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service] Service 'ArcGIS Notebook Server' already stopped, no action required.

[02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]: LCM:  [ End    Set      ]  [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service]  in 0.0310 seconds.

[02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]: LCM:  [ End    Resource ]  [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service]

[02/16/2021 15:41:48] [VERBOSE] [TAPORTAL-PRI]: LCM:  [ End    Set      ]

[02/16/2021 15:41:48] [ERROR] The SendConfigurationApply function did not succeed.

Scenario 2:

• Granted ArcGIS domain user the following roles in Azure Portal before running the Cloud Builder:

1. Contributor
2. Reader and Data Access
3. Storage File Data SMB Share Contributor

• Ran Cloud Builder with the same parameters similar to all my trials.
• Used domain join option with arcgis service account as a domain account and used Azure Cloud Storage for the configuration and content store.
• Didn’t do any manual setting changes while Cloudbuilder was running like the ones that I did in Scenario1

Result : Similar to Scenario 1 result

[02/15/2021 22:33:10] [VERBOSE] [taPortal-Pri]:                            [[ArcGIS_WindowsService]ArcGISGeoEventGateway_Service] User name for service 'ArcGISGeoEventGateway' is 'LocalSystem'. It does not match 'tca\arcgis.

[02/15/2021 22:33:10] [VERBOSE] [taPortal-Pri]: LCM:  [ End    Test     ]  [[ArcGIS_WindowsService]ArcGISGeoEventGateway_Service]  in 0.0160 seconds.

[02/15/2021 22:33:10] [VERBOSE] [taPortal-Pri]: LCM:  [ Start  Set      ]  [[ArcGIS_WindowsService]ArcGISGeoEventGateway_Service]

[02/15/2021 22:33:10] [VERBOSE] [taPortal-Pri]:                            [[ArcGIS_WindowsService]ArcGISGeoEventGateway_Service] Service 'ArcGISGeoEventGateway' already stopped, no action required.

[02/15/2021 22:33:10] [VERBOSE] [taPortal-Pri]: LCM:  [ End    Set      ]  [[ArcGIS_WindowsService]ArcGISGeoEventGateway_Service]  in 0.0310 seconds.

[02/15/2021 22:33:10] [VERBOSE] [taPortal-Pri]: LCM:  [ End    Resource ]  [[ArcGIS_WindowsService]ArcGISGeoEventGateway_Service]

[02/15/2021 22:33:10] [VERBOSE] [taPortal-Pri]: LCM:  [ Start  Resource ]  [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service]

[02/15/2021 22:33:10] [VERBOSE] [taPortal-Pri]: LCM:  [ Start  Test     ]  [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service]

[02/15/2021 22:33:10] [VERBOSE] [taPortal-Pri]:                            [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service] User name for service 'ArcGIS Notebook Server' is 'LocalSystem'. It does not match 'tca\arcgis.

[02/15/2021 22:33:10] [VERBOSE] [taPortal-Pri]: LCM:  [ End    Test     ]  [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service]  in 0.0150 seconds.

[02/15/2021 22:33:10] [VERBOSE] [taPortal-Pri]: LCM:  [ Start  Set      ]  [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service]

[02/15/2021 22:33:10] [VERBOSE] [taPortal-Pri]:                            [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service] Service 'ArcGIS Notebook Server' already stopped, no action required.

[02/15/2021 22:33:10] [VERBOSE] [taPortal-Pri]: LCM:  [ End    Set      ]  [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service]  in 0.0320 seconds.

[02/15/2021 22:33:10] [VERBOSE] [taPortal-Pri]: LCM:  [ End    Resource ]  [[ArcGIS_WindowsService]ArcGIS_Notebook_Server_Service]

[02/15/2021 22:33:10] [VERBOSE] [taPortal-Pri]: LCM:  [ End    Set      ]

[02/15/2021 22:33:10] [ERROR] The SendConfigurationApply function did not succeed.

Note: Each deployment was carried out afresh after deleting all the resources of the previous deployments using Cloud Builder and deleting the storage accounts manually in Azure Portal.

I am unable to fathom this out. Have escalated it to Esri support and am yet to hear from them on this error.

The only workable deployment that I am able to achieve is by carrying out deployment by not using 'Use Azure Cloud Storage for the configuration and content store?' option.

Will keep the community posted on the progress but in the meanwhile if anyone understands this issue and have any idea about it, kindly revert.

Regards,

Tejwant Kaur

[Anonymous User]

@TCAPublicAccount  (Tejwant)

Are your storage account and ArcGIS Enterprise deployment in the same region?

-Manoj

[TCAPublicAccount]

Hi Manoj,

Thanks for the response. Yeah, all the resources, including storage account are in same region.

Regards,

Tejwant Kaur

[DeeptiKushwaha]

I am also  getting same error  for accessing  Azure FS    while deploying  ArcGIS   enterprise .The FS  have been mounted on  al VMs and  validated  that  svc id    can access the azure  FS  with UNC  path  and can write to  FS . Its  a  multimachine  deployment  in azure using  ansible.  

messages":  [

                     "Failed to create the site. Configuration store error. Cannot get persistence. File system \u0027\\\\stappitsxxxxx.file.core.windows.net\\fs-xxxxagis-cus-917\\stageenterpriseprod\\server\\config-store\u0027 connection failed. The specified location is not accessible. Ensure that the ArcGIS Server account has read and write access to the location. "

                 ],

    "code":  500

}

[JoshHevenor1]

I'm having the same problem. I swear I had this working for another client 6 years ago by using "cmdkey /add: ..." to save the storage account key and referencing via UNC path. I had a link discussing this but misplaced it.  I've been trying aad domain accounts, and adding the key and still no dice.  I'm not using cloudbuilder, but ESRI PowerShell DSC.    

So, who's had this work? What users were performing what actions? Did you mount the Azure FS or use UNC path?