topic Re: Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups? in ArcGIS Enterprise Questions

Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups?

WhereMatters — Mon, 27 Jul 2015 12:47:35 GMT

ArcGIS DataStore schedules regular backups of its database every 7 days by default and retains them for 31 days.

Due to this, the disk space is getting filled up over a period of time.

I am also seeing that most of the disk space is occupied at "C:\arcgisdatastore\backup\walarchive".

I am thinking these are postgres db archive files as the name suggests.

Is it ok to clean up this directory completely from time to time?

Will this affect the ability to restore backups?

Re: Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups?

ChrisSmith7 — Mon, 27 Jul 2015 13:54:39 GMT

Anand,

Have a look at the following:

Manage data store backups—Installation Guides (10.3 and 10.3.1) | ArcGIS for Server

Here, you can find information on adjusting retention time, which can reduce disk space utilization:

Change how long backup files are retained
The backup directory retains backup files for 31 days by default. That means if you keep the default backup frequency (every 7 days) and retention schedules (31 days), the backup directory will contain five full backups and 31 days of incremental backup files. The size of these files depends on the amount and size of your data. The machine that stores your backups must have enough disk space for all these files. If you increase the backup frequency, consider decreasing the retention period for the backup files. In the previous section, the backup frequency was increased to every 14 days. To prevent your backup directory from becoming too large, decrease the backup file retention period.
The syntax to run the updatebackupretaindays utility is as follows:
updatebackupretaindays <number of days>
In the following example, backup file retention time is changed to 27 days:
updatebackupretaindays 27

On a side note for PostgreSQL installations, I've heard of some people using PostgreSQL: Documentation: 9.5: pg_archivecleanup for cleanup... I'm not sure if it's compatible with ArcGIS. Here's a discussion on better understanding what the tool will do:

postgresql - I need help understanding Postgres's archive cleanup functionality - Database Administrators Stack Exchange

Re: Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups?

JosephFrechette — Thu, 06 Aug 2015 16:47:58 GMT

Chris,

Does this change the retain days for the incremental backups that are in the walarchive folder too? Or does making these changes only affect the full dbbackup?

Re: Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups?

TonyGegner — Wed, 19 Aug 2015 11:23:21 GMT

Yea, the walarchive folder is an issue. Mine is making a new 7MB file every 5min which is a lot of space over time.

I didn't have a problem to delete files from it, deleted 36000 files yesterday.

Must be some way to change the interval on this as well.

Re: Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups?

ChrisSmith7 — Wed, 19 Aug 2015 14:31:55 GMT

I'm not sure, but it seems like it would affect incremental backups. According to the documentation:

By default, the data store creates a full backup of feature layers once a week and incremental backups either when the log files are full or every 5 minutes, whichever comes first. The database controls incremental backup creation, but you can change how often the data store creates a full backup by running the updatebackupschedule utility.

...if you use default backup settings, backups will contain five full backups and 31 days of incremental backup files

The backup directory retains backup files for 31 days by default. That means if you keep the default backup frequency (every 7 days) and retention schedules (31 days), the backup directory will contain five full backups and 31 days of incremental backup files.

If you ran something like updatebackupretaindays 27 it seems that it would then only keep 27 days of incremental back-ups and 4 full backups (assuming you kept the full backup frequency of 7 days).

Re: Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups?

JosephFrechette — Wed, 19 Aug 2015 15:53:47 GMT

I tested this out. These are the settings that I applied:

updatebackupschedule --starttime <local server time> --frequency 1

updatebackupretaindays 1

With these settings the incremental backup kept going until I deleted the files going back over a week. I don't think that the settings are being applied to the incremental backups. That walarchive folder gets large very fast regardless of the settings.

To conserve space I have been deleting the .gz files about once a week. This has not caused any issues.

Re: Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups?

ChrisSmith7 — Wed, 19 Aug 2015 16:00:54 GMT

It sounds like it should be applying the settings to the incremental back-ups - is this a bug, perhaps?

Re: Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups?

SohamShah3 — Thu, 06 Apr 2017 14:50:29 GMT

How do you manage the backups for portal? Walarchive folder in portal is also storing too many backups for me.

Re: Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups?

JonathanQuinn — Sat, 08 Apr 2017 01:12:45 GMT

You're using 10.5, correct? You can run a full backup for Portal which will clear out the walarchive files. It's a best practice to be backing up your Portal anyway. This can be done using the DR tool, (recommended for full ArcGIS Enterprise deployments) or through Portaladmin.

Re: Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups?

KirkChesla — Tue, 21 Nov 2017 18:28:55 GMT

For our brand new Portal installation on RHEL7 the walarchives have completely filled up our /opt directory in just a few weeks of idle use. And now Portal will not start up even after clear some space. Then the webgisdr shell script will not run at all from Putty.

Its starting to look like another re-install of Portal.

Re: Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups?

JonathanQuinn — Wed, 22 Nov 2017 18:48:57 GMT

I'm not sure if you've started the uninstall/reinstall yet but have you checked the service error logs under <install dir>/portal/usr/arcgisportal/logs/service.log? They may give some insight as to why Portal doesn't start.

Re: Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups?

Todd_Metzler — Thu, 04 Jun 2020 17:02:20 GMT

Is it possible to use the DR Tool to back up Enterprise 10.4.1 (federated portal/server no Data Store) and then recover this backup to fresh installation of 10.7.1 Enterprise single stack (portal/server/data store). The past three times (10.5.1 > 10.6.1 > 10.7.1) I have run the upgrade installer on single stacks it has failed with catastrophic results to the portal DBs and I don't trust the process. the 10.4.1 Enterprise is my production environment and I can't afford to kill it off and do a fresh install nor hold off upgrade any longer to

Re: Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups?

JonathanQuinn — Thu, 04 Jun 2020 17:36:04 GMT

No, the DR tool backups are not forward compatible. We do plan on supporting that at some point, but even that workflow would internally trigger an upgrade.

Have you reached out to Support regarding your failed upgrade attempts? Do you recall if they all cite similar errors?

Re: Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups?

Todd_Metzler — Thu, 04 Jun 2020 18:12:05 GMT

Thank you for the speedy reply. Yes, reached out to support via case when failure happened (10.6.1 > 10.7.1). I don't recall if the failure point was exactly the same. I do recall the installer logs not being very helpful. After 24 hours (3 work days) of investment in attempting to migrate Portal DBs manual process following steps provided by support gave up and did fresh install. Thus I don't trust the upgrade installer.

Re: Is it safe to delete files in 'walarchive' from ArcGIS DataStore backups?

JonathanQuinn — Fri, 05 Jun 2020 16:04:04 GMT

Do you have a standby or dev environment to upgrade, or are you upgrading your production environment? The only way to get to a newer version is to upgrade. We put together a blog on minimizing downtime and risk during an upgrade:

https://www.esri.com/arcgis-blog/products/arcgis-enterprise/administration/minimize-downtime-when-upgrading-arcgis-enter…

You can use this approach to vet out the upgrade and work through any issues without impacting production.