topic Re: ArcGIS Enterprise 10.6 Data Store: folder pg_xlog keeps increasing 16MB every 5min in ArcGIS Enterprise Questions

ArcGIS Enterprise 10.6 Data Store: folder pg_xlog keeps increasing 16MB every 5min

BrianCoffey2 — Mon, 23 Apr 2018 20:05:25 GMT

ArcGIS Enterprise 10.6 Data Store: folder pg_xlog keeps increasing 16MB every 5min

(ArcGIS Enterprise Portal, Server and Data Store are all installed on E: drive)

I setup Data Store with a domain account and checked all the folders under E:\arcgisdatastore and E:\ArcGIS\DataStore. The domain account has full control permissions in all folders. The Data Store service is running as the domain account. In the postgresql.conf file the archive_command is as follows:

archive_command = 'E:/ArcGIS/DataStore/framework/etc/scripts/dbcli put-transaction-log "E:/arcgisdatastore/pgdata/%p" "%f" "E:/ArcGIS/DataStore"'

This is the error message I see in the folder E:\arcgisdatastore\logs\machine name\database, see attachment.

How do I fix?

Re: ArcGIS Enterprise 10.6 Data Store: folder pg_xlog keeps increasing 16MB every 5min

Anonymous User — Mon, 23 Apr 2018 21:02:29 GMT

I'd check the following:

1. Confirm the service account running the ArcGIS Data Store.

2. Once you have that account running the ADS confirmed, check the folder on the E:\ drive for explicit permissions. It's possible your IT prevents the option to cascade permissions to subfolders.

3. I'd "reforce" permission on the ADS parent directory (read/write) --> go to advance and chose the option to 'enable parent inheritance' for the permissions.

The 'Access is Denied' error in the logs seems to be the culprit; ensuring explicit permissions may be a simple fix if that is in fact the issue.

Re: ArcGIS Enterprise 10.6 Data Store: folder pg_xlog keeps increasing 16MB every 5min

BrianCoffey2 — Tue, 24 Apr 2018 13:27:47 GMT

Verified the service is running under the appropriate domain account. Forced permissions on folders E:\ArcGIS\DataStore and E:\arcgisdatastore and their subfolders and files. The recurring logs keep writing 16MB files every 5min. Attached are pics of the verifications of the actions described above. At a loss???

Re: ArcGIS Enterprise 10.6 Data Store: folder pg_xlog keeps increasing 16MB every 5min

Anonymous User — Tue, 24 Apr 2018 13:35:46 GMT

Are you able to upload (or send directly to me, if you prefer) one of the complete logs?

Re: ArcGIS Enterprise 10.6 Data Store: folder pg_xlog keeps increasing 16MB every 5min

BrianCoffey2 — Tue, 24 Apr 2018 13:50:01 GMT

I have attached the last couple of days of logs

Re: ArcGIS Enterprise 10.6 Data Store: folder pg_xlog keeps increasing 16MB every 5min

BrianCoffey2 — Mon, 30 Apr 2018 13:02:50 GMT

Also this ArcGIS Enterprise install is on a Windows 2016 Server machine.

Re: ArcGIS Enterprise 10.6 Data Store: folder pg_xlog keeps increasing 16MB every 5min

ClintonBallandis1 — Mon, 05 Aug 2019 03:15:27 GMT

Hi Brian,

Did you manage to get the log file issue resolved ?

Thanks,

Clinton

Re: ArcGIS Enterprise 10.6 Data Store: folder pg_xlog keeps increasing 16MB every 5min

JoshuaDalton — Tue, 05 Nov 2024 23:10:52 GMT

An old topic, but I'm having similar issues with an ArcGIS Datastore 11.3. Oddly we had determined that pgdata\pg_wal would fill with the files if ArcGIS Datastore was running as a domain account with administrator access and Full Control on the whole drive and all subfolders, and you would get the error messages:

2024-10-19 00:02:17 EDT: [2844]: [2495-1] LOG: archive command failed with exit code 1
2024-10-19 00:02:17 EDT: [2844]: [2496-1] DETAIL: The failed archive command was: E:/Apps/ArcGIS/DataStore/framework/etc/scripts/dbcli put-transaction-log "E:/arcgisdatastore/pgdata/pg_wal\000000010000000000000002" "000000010000000000000002" "E:/Apps/ArcGIS/DataStore/"
Access is denied.

But... it would archive the files out over time if you stopped the service and made it run as Local System.

I'm not sure how that reflects on the fix I made though. To fix it, I noticed the single \ in the archive command used. This is coming from using %p in pgdata\postgres.conf under the archive_command setting. Since some older postgres issues were documented elsewhere on the web with regard to it interpreting special characters in the archive_command, I changed the command setting from this:

archive_command = 'E:/Apps/ArcGIS/DataStore/framework/etc/scripts/dbcli put-transaction-log "E:/arcgisdatastore/pgdata/%p" "%f" "E:/Apps/ArcGIS/DataStore/"'
to this:
archive_command = 'E:/Apps/ArcGIS/DataStore/framework/etc/scripts/dbcli put-transaction-log "E:/arcgisdatastore/pgdata/pg_wal/%f" "%f" "E:/Apps/ArcGIS/DataStore/"'

Now it seems to be archiving. I'm not sure if other folders may try to also use this command, but it definitely got it to stop leaving 16mb files in the pg_wal folder every 5 minutes. I'll watch logs and see if anything else fails as a result of the change.

Re: ArcGIS Enterprise 10.6 Data Store: folder pg_xlog keeps increasing 16MB every 5min

JoshuaDalton — Thu, 17 Apr 2025 19:17:19 GMT

And back to this error again on yet another machine on 11.1. In addition to changing %p to pg_wal/%f, we added double-quotes around the path for the dbcli command, added the .bat extension (probably unnecessary), and since quoted, we removed the ^ from the Program Files sections (both places), which changed the archive_command from this:
archive_command = 'D:/Program^ Files/ArcGIS/DataStore/framework/etc/scripts/dbcli put-transaction-log "D:/arcgisdatastore/pgdata/%p" "%f" "D:/Program^ Files/ArcGIS/DataStore/"'
to this:
archive_command = '"D:/Program Files/ArcGIS/DataStore/framework/etc/scripts/dbcli.bat" put-transaction-log "D:/arcgisdatastore/pgdata/pg_wal/%f" "%f" "D:/Program Files/ArcGIS/DataStore/"'

This was the entry in the logs before applying the changes:

2025-03-25 00:02:55 EDT: [12304]: [105467-1] LOG: archive command failed with exit code 1
2025-03-25 00:02:55 EDT: [12304]: [105468-1] DETAIL: The failed archive command was: D:/Program^ Files/ArcGIS/DataStore/framework/etc/scripts/dbcli put-transaction-log "D:/arcgisdatastore/pgdata/pg_wal\000000010000000000000002" "000000010000000000000002" "D:/Program^ Files/ArcGIS/DataStore/"

I can confirm that the dbcli.bat file was not launching before our change, as we watched for it in ProcessMonitor. So this time it seems less like a source/target folder permission, and more like permission to launch the dbcli command in the way it is being passed to the windows command line shell by postgres.

We did try using D:\Progra~1\... instead of D:\Program^ Files\... but that did not work without the quotes. Leaving the ^ in with quotes also still gave the error. Possible the ~1 may work with quotes.

Long story short, probably best not to install to folders with spaces in them, and correcting the %p with pg_wal/%f is still also a good idea. Quoting the dbcli command and removing the ^ escape characters seems to correct the issue when a space exists in the path.