Latest Contributions by NicolasGIS

Thanks ! In the meantime, I am currently trying to restore on a drive with higher IO. Yes, the issue is occuring while trying to restore a webgisdr created with "BACKUP_RESTORE_MODE = full" ... View more

By the way, the "TOKEN_EXPIRATION_MINUTES" of the webgisdr is very confusing and poorly documented ! ... View more

It's done on 10.9.1 fully patched (including "Durability Enhancement Patch" that is supposed to improve tileCache performance...) It's really bad news this value is hard-coded ! Please consider externalizing it as we are at the very beginning of our use of the tileCache with the BIM becoming more and more a thing. It's scary to think we are already at the limit ! Will do some more testing based on your feedback... for a change... ... View more

FYI@JonathanQuinn, I was able to solve my issue which seems to be related to the disk performance I was restoring the datastore on. On the drive it was failing, it has as characteristics: 80 MB/s and 100 IO Operations (both read and write) I tried on a "better" drive with: 300MB/S and a rate of 5 IO operation per gigabyte with a guaranteed minimum of 500 IO operations and a maximum of 2000 IO operations (both, read and write) and the restore worked 🙂 ... View more

My tilecache is now taking more than 24hours to restore and the job is failing because of a job timeout: 2022-09-16 11:12:33 ERROR [pool-1-thread-2] com.esri.arcgis.webgis.component.service.impl.DataStoreDRService - {"jobId":"653c93cb-e0bd-475e-8120-87eae7f61f3a","description":"Deploy data store snapshot 20220910-180921-CEST-86-FULL from \\\\PATHTO\\backup\\temp\\WebGISSite1663222328934\\dataStore\\a3729eb8-a53e-4ac5-88a4-d90a56801b3b","lastModified":"2022-09-15 11:12","status":"scheduled"} 2022-09-16 11:12:33 INFO [main] com.esri.arcgis.webgis.util.WebGISUtil - The restore of ArcGIS Data Store has taken 24hr:00min:04sec. 2022-09-16 11:12:33 ERROR [main] com.esri.arcgis.webgis.service.impl.WebGISDRDispatcher - Exception: Failed to restore the ArcGIS Data Store. 2022-09-16 11:13:05 ERROR [main] com.esri.arcgis.webgis.client.WebGISDR - Failed to restore the ArcGIS Data Store. Any idea how could I increase this threshold ?  I am under the impression that it's not a classic ArcGIS Server geoprocessing job (I could not find any jobs related to that in arcgisserverjobs directories) so don't know where to look. I don't think it is related to the "TOKEN_EXPIRATION_MINUTES" property of the webgisdr as I did not modified it and so it must be configured to the default 60 minutes which obviously is not the case so I don't know what is this property used for... Only for ArcGIS Server sites ? https://enterprise.arcgis.com/en/portal/latest/administer/windows/create-web-gis-backup.htm   @JonathanQuinn  or @ChristopherPawlyszyn  maybe ?   Thanks ! ... View more

Done ... View more

Good news the patch is now available: ArcGIS Data Store 10.9.1 Durability Enhancement Patch  .. but it only partially solved my issue of restore unfortunately ☹️ I no longer have the error: "No valid connection to ArcGIS Data Store configuration store established." is my logs but I still have "ArcGIS Data Store has detected an issue with 'db'."... and the restore fails. <Msg time="2022-09-06T17:17:12,981" type="WARNING" code="110130" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg> <Msg time="2022-09-06T17:17:35,106" type="WARNING" code="110379" source="Data Store" process="6652" thread="23" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Failed to restore 'relational' data store. Failed to start relational data store.</Msg> <Msg time="2022-09-06T17:17:49,13" type="WARNING" code="110130" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg> <Msg time="2022-09-06T17:18:27,233" type="WARNING" code="110130" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store has detected an issue with 'db'.</Msg> <Msg time="2022-09-06T17:18:27,233" type="SEVERE" code="110131" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">ArcGIS Data Store encountered too many problems. Failover may be invoked if standby is configured.</Msg> <Msg time="2022-09-06T17:18:28,280" type="SEVERE" code="110453" source="Data Store" process="6760" thread="1" methodName="" machine="GISSTOREREC01.COMPANY.COM" user="" elapsed="" requestID="">Data store machine 'GISSTOREREC01.COMPANY.COM' has failed.</Msg> The restore seems to be working fine in the logs from what I can tell but afterward, when starting DB, the following is logged: 2022-09-06 17:17:39 CEST: [2280]: [1-1] FATAL: the database system is starting up 2022-09-06 17:17:39 CEST: [92]: [4-1] LOG: redo done at 6/2FFB6D18 2022-09-06 17:17:41 CEST: [5556]: [1-1] FATAL: the database system is starting up 2022-09-06 17:17:41 CEST: [1832]: [1-1] FATAL: the database system is starting up 2022-09-06 17:17:42 CEST: [92]: [5-1] FATAL: could not write block 0 in file "base/17591/1257285": No such file or directory 2022-09-06 17:17:42 CEST: [92]: [6-1] CONTEXT: writing block 0 of relation base/17591/1257285 2022-09-06 17:17:43 CEST: [4584]: [6-1] LOG: startup process (PID 92) exited with exit code 1 2022-09-06 17:17:43 CEST: [4584]: [7-1] LOG: aborting startup due to startup process failure 2022-09-06 17:17:43 CEST: [4584]: [8-1] LOG: database system is shut down The funny thing is that on production datastore, there is no "base/17591" folder either so I don't know why does it try to write in this file... I guess I am good for another loop with support now ! Anyway, @JonathanQuinn  you can thank your team for the patch ! It's much appreciated.  ... View more

Same for the one online @George_Thompson :   NB: I am using 10.9.1 ... View more

Thanks for your reply Christopher. In the end, I ended up recreating the ArcGIS Enterprise deployment from scratch at 11.0 and importing the webgisdr backup.  Many thanks for your precious support and good avices ! You had a good intuition from the beginning that hostname.properties file was involved ! Thanks ... View more

Hello @JonathanQuinn , I am coming back to you regarding the patch for this bug. I was told it would be available by mid-August, but it is still not the case. Any information about that on your side ? Thanks ! ... View more

So I commented out the configuration of the hostname.properties file in ArcGIS Upgrade section as well as you suggested: https://github.com/Esri/arcgis-powershell-dsc/blob/master/Modules/ArcGIS/DSCResources/ArcGIS_ServerUpgrade/ArcGIS_ServerUpgrade.psm1#L73-L80 And it now works ! So the root cause is now understood: great 🙂   Many many thanks for your support to identify the root cause !   Do you think there could be a way to "fix" a corrupted ArcGIS Server site ? What would be your recommandation ? Thanks ! ... View more

Well spotted. I was about to test 😉. Let me give it another run with this modification and I come back to you ! Thanks ! ... View more

FYI, @ChristopherPawlyszyn, I tried 2 additional tests: - Manual install of 10.9.1 -> webgidr export -> webgisdr import -> OK -> Manual upgrade to 11.0 -> webgisdr export -> webgisdr import -> OK - DSC Install of 10.9.1 -> webgidr export -> webgisdr import -> OK -> Manual upgrade to 11.0 -> webgisdr export -> webgisdr import -> OK So it looks like the issue only occur when upgrading using DSC. Does it help ? What could DSC be possibly doing differently than a manual install that would prevent ArcGIS Server from importing the site backup ?! I will make it crash once again, and search any evidence of a machine name change to answer your question.   Thanks ! ... View more

Thanks @ChristopherPawlyszyn  for your reply and time. It's really weird... Following your reply I tried again from scratch with the same result: ArcGIS Server fails to restore. In ArcGIS Server logs, I have plenty of errors such as: akka.pattern.AskTimeoutException: Recipient[Actor[akka://akka-rediscovery/user/AkkaAppServerActor#-1696377499]] had already been terminated. Sender[null] sent the message of type "com.esri.arcgis.discovery.admin.rediscovery.util.Message". But I don't think it is the root cause. The only difference I see between you and I, is that I use my own fork of ArcGIS DSC because of issue 132: https://github.com/Esri/arcgis-powershell-dsc/issues/132 I prevent DSC from configuring the hostname:   I had open a BUG regarding this issue: https://my.esri.com/#/support/bugs/BUG-000124901   As you asked about the hostname.properties file, do you think it could be related ? Regarding the issue itself, I don't know where to look for. Any idea what I could do to troubleshoot this issue ?   Finally, to answer to your question, the upgrade was done with Powershell DSC.   Thanks !   ... View more