"Wal files size exceeded the max size."

[MichaelSchoelen]

Every 60 seconds, our portal is delivering a warning that:

"Wal files size exceeded the maximux size." (not "Maximum"). Code: 218018Process: 5676

From our initial search, "Wal" might mean "Write Ahead Logging", which indicates that this could be a mis-configuration with the datastore? But we didn't find any unusually large files there.

We took a look at our hosting server, and it is delivering an error:

Error deleting WAL backup. Write to configuration store failed.. Extended error message: An I/O error occurred while sending to the backend.

We are running ArcGIS Enterprise 10.5.1 on Windows 2008 R2.

Ideas on what to look for? 

[RenaudJOGUET2]

Hi Michael,

Same problem here on Portal 10.5.1 on Redhat.

I solved my problem with this command line (direct on the machine in this directory):

[root@...]# rm -f /.../esri/arcgis/portal/usr/arcgisportal/backup/walarchive/*

(I do this every 14days.)

I think you can customize your backup configuration, as describe here :
Create an ArcGIS Enterprise backup—Portal for ArcGIS (10.6) | ArcGIS Enterprise 

Or

ArcGIS Data Store command utility reference—Portal for ArcGIS (10.5.x) | ArcGIS Enterprise 

But, I could not find any configuration to resolve my problem...

Interested in another solution 🙂

Thx,

Renaud

[JonathanQuinn · Accepted solution]

Those are errors indicating that the transaction log size for portal's internal database is greater than 5 GB. The proper way to get rid of them is to run a backup with the WebGIS DR tool, which also gives you a backup of your Enterprise, (always a good thing to schedule).

There is a 5 GB cap on the logs prior to running the DR tool. After you run it, the cap is removed so they can grow unrestricted. We're improving that process for future releases but for now, the ideal solution is to schedule backups, but you can also delete them manually if you're not using the DR tool to create incremental backups.

[MichaelSchoelen]

JQuinn-esristaff, So we are already running the WebGISDR tool for incremental backups every week, and a full backup every month. Would you suggest increasing the frequency of incremental backups, then?

[JonathanQuinn]

Only the full backup will clear out the transaction logs, as they're necessary for incremental backups.

Right now, it sounds like you find a weeks worth of data loss acceptable, considering you're backing up your enterprise once every week. If you ran an incremental backup every day and a full backup once a week, that will reduce the amount of potential data loss and get rid of the error you're seeing.

[SebastianStey]

We've got a portal 10.5.1 in high availability configuration (portal consists of two machines).

We scheduled webgisDR full backups on a daily basis. Thus, we expected the walarchive folder to be emptied every day. However, this is the case only for the primary portal machine. The walarchive folder on the standby portal machine remains unchanged and we receive the log message about exceeded maximum size every minute.

1. Does this indicate, the standby machine is not taken into account when the backup is created? Will that mean any problems for restoring?

2. How to make sure a webgisDR full backup also empties the standby's walarchive Folder?

[RamakrishnaBillakanti]

@Jonathan Quinn Hi Jonthan, What directory these WAL files are created. I'm using 10.6 Federated Enterprise Environment. I can't take the backups using the WebGISDR Utility due to a bug with the tool with IWA Integrations. Can you point me the folder where I need to delete the WAL files to resovle the issue with warning messages. I've ample space on the hard drive but still seeing these errors. Thank you in advance. 

[JonathanQuinn]

I'm familiar with the IWA bug, but using the Portal URL directly, (https://portal.domain.com:7443/arcgis) for the PORTAL_ADMIN_URL should work. Did you try that but still see an issue? If so, can you append sharing/rest to the end, (ex. https://portal.domain.com:7443/arcgis/sharing/rest) and see if it redirects to the URL that is secured with IWA?

The WAL archive logs are under the default content directory, for example C:\arcgisportal\backup\walarchive.

[RamakrishnaBillakanti]

Thank you Jonathan, Removing the files from Wararchieve back up directory worked. Really appreciate your quick response.

[RamakrishnaBillakanti]

Hi Jonathan,

This issue is repeating again in the portal, having removed the WAL files from the directory resolved the issue but do you know how this could be resolved permanently so this wont repeat again.