topic Re: Portal for ArcGIS walarchive in ArcGIS Online Questions

Portal for ArcGIS walarchive

DanielIrwin — Fri, 05 Oct 2018 09:04:03 GMT

Hi there,

So, asking for a friend, in the install directory for Portal there's a folder under <install dir>\backup\walarchive\ which seems to contain a backup file for every hour since it was installed. They've run out of disk space (each file is 16Mb, there's thousands of them), and it's now breaking their portal. So questions are (a) what is this backing up as there's a separate walarchive for the data store (i.e. it's not this one) and (b) how do you manage that backup? I've seen some stuff on managing the backup of the data store, but that doesn't seem to affect this backup?

I'm guessing it's ok to delete the older ones, but they would prefer no to have to keep going back to them over and over again...

Many thanks in advance

Dan

Re: Portal for ArcGIS walarchive

JonathanQuinn — Fri, 05 Oct 2018 20:21:12 GMT

Those files exist to support incremental backups with the DR tool. Prior to running the DR tool to create a full backup, the directory is capped at 5 GB. Once you run the DR tool, that limit is removed. If you run the DR tool to create a new full backup, the directory will be cleaned out. We incorrectly assumed customers would run the DR tool at scheduled times, which would manage the size of the directory. Once the space is cleared up and Portal is back to working order, you can either run the DR tool on a scheduled task to create backups (recommended), or, if you're not going to run incremental backups, delete the backups manually.

Re: Portal for ArcGIS walarchive

DanielIrwin — Mon, 08 Oct 2018 09:58:28 GMT

Hi Jonathan

Many thanks, I have passed this information on to the person who was asking, I'm not sure what their DR policy is, hopefully this can solve it for them!

Regards

Dan

Re: Portal for ArcGIS walarchive

pfoppe — Tue, 19 May 2020 16:52:44 GMT

Hi Jonathan Quinn‌,

We have a situation where we are running weekly full backups (using webgisdr) and daily incrementals, but the C:\<portal_dir>\backup\walarchive folder is not being cleaned up after a webgisdr full backup. This is chewing up the available C:\ space and we have been manually purging some of these files that are over 60 days.

Per the Common problems and solutions—Portal for ArcGIS (10.8) | Documentation for ArcGIS Enterprise The portal content directory has grown to several gigabytes in size

Once you run a backup using the webgisdr tool, this limit is removed, but the transaction logs are cleared each time the tool is run.

We have a few ArcGIS Enterprise deployments, and this is the only one causing problems. It is also the only environment we (unsuccessfully) attempted a portal content C:\ to D:\ migration... so I suspect the situation is that the webgisdr utility is trying to purge the D:\<portal_dir>\backup\walarchive instead of the C:\ location... thus leaving these behind for us to have to clean up manually. GeoNet reference to that thread - https://community.esri.com/message/904321-re-moving-arcgis-portal-directories#comment-929961

Thoughts on this issue? I suspect the suggestion will be to open a new support case (referencing the support case and bug logged on the referenced thread).

Thanks!

Leeah Risher‌

Re: Portal for ArcGIS walarchive

JonathanQuinn — Thu, 21 May 2020 17:19:01 GMT

Foppe, Patrick‌ thanks for the feedback, looks like there are a few compounding issues here. Was Support also able to reproduce the behavior you saw in regards to the transaction logs not getting cleared?

Re: Portal for ArcGIS walarchive

pfoppe — Thu, 21 May 2020 17:24:48 GMT

Hi Jonathan Quinn‌,

We have not opened an esri support incident on this yet, we actually discussed doing this today. Esri support was able to reproduce the issues of the C:\ to D:\ content migration (referenced above) and I suspect this is related to that.

When we start to run out of C:\ space, we manually purge some of these walarchive files to keep the system operational. Thanks

Re: Portal for ArcGIS walarchive

Anonymous User — Thu, 02 Jul 2020 03:52:25 GMT

Hi Jonathan,

I did notice our STAGING is not clearing up walarchives but PROD is clearing it well and keeping only for current date.

We have webgisdr in place and we are doing full + incremental backups as suggested by esri for both environments, we are also using 10.8 version of esri. What could be the reason?

Re: Portal for ArcGIS walarchive

DavidHoy — Fri, 16 Oct 2020 01:25:06 GMT

Jonathan Quinn‌

Is there a way to reset the original 5GB limit on the size of the walarchive directory for the cases where webgisdr export was run once but not subsequently?

Re: Portal for ArcGIS walarchive

JonathanQuinn — Mon, 26 Oct 2020 19:30:41 GMT

No, you'll need to run the DR tool or delete the files manually (except for the latest one). We are working on a better story behind those transaction logs.

Re: Portal for ArcGIS walarchive

JanSarata — Tue, 09 Feb 2021 14:07:02 GMT

Hi @JonathanQuinn ,

is there any update regarding this behaviour?

Is there a way how completely turn these incremental backups off? I would like to get rid of this feature since the customer has different way to make backups (daily backup of virtual images).

Appreciate any new update.

Have a good day,

Jan

Re: Portal for ArcGIS walarchive

BillLotz — Thu, 08 Apr 2021 14:00:37 GMT

Looking for updates as well. The ability to turn it off would be nice, or to reinstate the cap.

Re: Portal for ArcGIS walarchive

AnibalMmartinez — Thu, 17 Jun 2021 15:56:20 GMT

Tenemos el mismo problema en versión 10.7.1 y no entendemos la solución que propones, ¿podrías indicar paso a paso que hay que hacer, como frenamos este incremento de archivos permanentes, sin eliminar archivos manualmente?

Saludos,

Anibal Telecom Argentina

Re: Portal for ArcGIS walarchive

lvargas — Thu, 17 Jun 2021 16:12:40 GMT

Hola @AnibalMmartinez

A partir de la versión 10.5, la base de datos interna del Portal (diferente del ArcGIS Data Store) hace una copia de seguridad en \arcgisportal\backup\walarchive. Estos registros walarchive registran mucho más que las transacciones reales dentro de la base de datos. Registran el estado de la base de datos y otras actividades internas no visibles para el administrador o el usuario del portal. Se escriben en el disco cada hora (16 MB aprox.), independientemente de la actividad; el ritmo de escritura puede ser superior a una hora en algunos casos. Existe una restricción del espacio utilizado pero la misma se quita cuando se realiza un respaldo completo, o bien se hace una carga de un respaldo previo en el ambiente.

En la actualidad no hay manera de controlar las copias de seguridad incrementales de Portal (en versiones previas a 10.9)*. Estas son copias de seguridad para proporcionar la información necesaria cuando se realice una copia de seguridad completa; la manera de borrar estos registros es ejecutar un webgisdr completo (un respaldo de la configuración y los datos del ambiente).

*Nota: La capacidad de modificar la copia de seguridad incremental de Portal esta disponible para ArcGIS Enterprise 10.9.

Teniendo claro lo anterior, las opciones que se pueden abordar son:

Verificar el espacio en el disco duro del equipo en donde está el Portal for ArcGIS para validar que dicho servidor no tenga problemas de espacio.
Una vez descartado lo anterior, ejecutar de forma periódica un respaldo completo de la configuración y los datos del ambiente a través de la utilidad webgisdr (esto para borrar los registros de transacción del Portal) Se debe de tener una ventana de mantenimiento en la cual no se realicen modificaciones al ambiente; se podría hacer de forma manual o bien programando una tarea automática en el servidor.
Eliminar los archivos con cierta periodicidad de forma manual (manteniendo al menos 1 día de respaldos incrementales).
Programar un script (un bat, python etc.) para que de forma automática elimine estos archivos (manteniendo al menos 1 día de respaldos incrementales)

Espero que los datos sean de utilidad.

Saludos.

Re: Portal for ArcGIS walarchive

AnibalMmartinez — Thu, 17 Jun 2021 18:46:12 GMT

Mil Gracias @lvargas !

Re: Portal for ArcGIS walarchive

SanderBroekaert — Tue, 17 May 2022 14:56:40 GMT

Do you have an update on this? We are on ArcGIS Enterprise 10.7.1...

Re: Portal for ArcGIS walarchive

DavidHoy — Wed, 25 May 2022 01:10:05 GMT

At 10.9+, a backup mode has been introduced to webgisdr. This mode re-instates the 50MB limit on transaction logs if it has previously been disabled. If your environment doesn't require geographic redundancy where you restore to a secondary data center, set BACKUP_RESTORE_MODE to backup instead of full. (Full will still clean out ALL files in walarchive)
This backup mode can be used when limited hard drive space is available on the portal machine or when backups are not regularly scheduled. When using this mode, the default size limit is enforced on internal transaction logs, preventing them from filling up your hard drive. If you only create full backups, you can set this property to backup.

Re: Portal for ArcGIS walarchive

TimHayes3 — Fri, 02 Dec 2022 19:38:52 GMT

When you mention "if you're not going to run incremental backups, delete the backups manually", by manually do you mean go into File Explorer walarchive folder select all the files in this folder then hit delete?

Re: Portal for ArcGIS walarchive

DavidHoy — Tue, 06 Dec 2022 04:16:09 GMT

yes - simple delete in windows explorer of files older than your last full backup is safe

Re: Portal for ArcGIS walarchive

Trevor_Hart — Mon, 24 Apr 2023 19:02:09 GMT

Still happening at 10.9.1. I dont use WebGISDR on my development system and its always running out of space.

I wrote this batch file which will clear out the walarchive folder using the pg_archivecleanup Postgres utility

off SET PGBIN=C:\ArcGIS\Portal\framework\runtime\pgsql\bin SET LOC=C:\arcgisportal\backup\walarchive FOR /F "delims=|" %%I IN ('DIR "%LOC%" /B /O:D') DO SET NewestFile=%%I echo %NewestFile% %PGBIN%\pg_archivecleanup.exe %LOC% %NewestFile% pause