ArcGIS Server on Windows Server 2022 high disk usage

[FentonCheng_-_DIT]

Has anyone seen unexplained high disk usage with ArcGIS Server 11.1 on Windows Server 2022 Datacenter? This is a standalone server, not federated to any Portals.

Our server monitoring has picked up 2 times where the server storage has spiked to over 90% space used (normal for the registered folders is around 30% used). No new services are being published and service usage is low. No tile cache creation jobs are in progress (this is intended to be an internal basemap server). Web adaptor is on a different machine.

Logging onto the server does not show any large files being created, but File Explorer shows the drive as running out of space. This is resolved by either restarting the main ArcGIS Server service or doing an OS restart.

[Scott_Tansley]

The main reason I've seen intermittent high disk usage is when someone is using a sync operation on the GIS Server to create a replica package for offline working.  That doesn't sound like the case if you're using it purely for tile cache though. 

Are your prepared cache/tiles on the same machine or stored remotely?  Do you burn cache on demand or is it 100% processed ahead of time?

Alternatively, what are your settings on page files?

[FentonCheng_-_DIT]

Tile caches are on the same machine and are 100% processed ahead of time (usually zoom levels 10-19). Tile cache jobs depend on the size and past behavior (ex: it's known that one of the older basemaps in use takes 3 hours to create all tiles at all zoom levels 10-19, so that one can run after hours). Larger jobs will run on a different machine and are copied over.

Page file: using system defaults (which may be an issue, the server has 128 GB memory)

[MichaelJenkins]

I got a low disk space warning not too long ago and found over 150,000 temp files from ArcGIS Server going back over 3 years.   I was able to delete many of them and freed up about 100GB of space on the C drive.

This article was helpful:  How To: Delete ArcGIS Enterprise Temporary Files (esri.com)

It has info that applies to stand-along ArcGIS Server as well as other Enterprise components.

For example, it shows this as one location where temp files can accumulate.

C:\Users\<user>\AppData\Local\Temp

Where <user> is the account that runs the ArcGIS Server service.

Also very helpful in these situations is a little program called "TreeSize Free" which you can install on your server and run it to see the names and size of all the folders in your drive tree, including hidden folders that you don't normally see in File Explorer.  

Another time we got a low disk space warning I used TreeSize Free and found another user had downloaded dozens of GBs of installation files into their "Downloads" folder, which is in their user profile directory and not one I would normally see or think to check.  

[FentonCheng_-_DIT]

I use WinDirStat for that, should look into TreeSize Free also.

Do temp files wind up in other locations? Server was installed into C, but the arcgisserver directory is on D (also the same drive that has the low disk space warning).

[A_Wyn_Jones]

@FentonCheng_-_DIT Has Windirstat identified what's consuming the disk space on D:\?

Worth checking that your caching tools aren't mistakenly running (check in Server Manager whether the caching tools have an instance in use).

[FentonCheng_-_DIT]

Caching tools are not running.

Last time this happened, WIndirstat found something but I didn't make a note of what was found at that time. I'll need to wait for the next time it happens, seems to be about once a month so far (not Windows Updates, that is handled by WSUS and it wasn't the day those are applied).