topic Re: arcgisinput folder: What is it for? in Publishing and Managing Services Questions

arcgisinput folder: What is it for?

jcarlson — Mon, 02 Aug 2021 18:11:47 GMT

I'm doing some maintenance on our Enterprise machines, and I happened to notice that some of the directories in our ArcGIS Server machine were not clearing out automatically. This meant that the output folder was getting quite large, when it should have been dropping files older than 10 minutes.

While I was dealing with that, I noticed that the directory arcgisinput was also quite large. According to the description, this folder

Stores source files (such as maps and data) for services that you publish.

It also appears to have no cleanup mode defined, which would imply that this folder is not meant to be cleared out. But I don't really understand what these source files are doing, if anything, and can't find any good documentation that explains it in more detail.

Key question: if I remove these files, will anything be impacted?

Most of our services are hosted. The Portal machine holds all the service definition files used to publish the services, so is anything using the raw files in the input directory?

In particular, I'm thinking about a few hosted imagery tile layers we have published. The service.sd file is about as big as the corresponding directory in arcgisinput. If I tell the service to build tiles, isn't it looking at the service definition when it does that?

It just seems like keeping these files as well as the service definitions leads to unnecessary duplication and requires extra storage space.

Re: arcgisinput folder: What is it for?

JenniferBorlick — Mon, 02 Aug 2021 19:57:31 GMT

I'm not sure if we are using it right, but this is where we put our map documents from which our map services are derived. Deleting them won't mess up what's already there, but it would be difficult to recreate or edit the services without those map documents there.

Re: arcgisinput folder: What is it for?

jcarlson — Mon, 02 Aug 2021 20:01:19 GMT

Our services are mostly "one and done", though the source projects and datasets are kept on our local network, just in case we never needed to reference them.

Are there instances when it has been useful for you to recreate/edit the service using those files?

Re: arcgisinput folder: What is it for?

JenniferBorlick — Mon, 02 Aug 2021 20:24:25 GMT

Pretty much every time we get a new server. We've been doing this since ArcIMS days, so I've used them probably about 5 times. I copy them over to the new machine and then open them and change the paths to the new server. (And do some updates while I'm in there.

Re: arcgisinput folder: What is it for?

DavidColey — Tue, 03 Aug 2021 16:00:46 GMT

So the input folder is used by server to store source copies of the services that you publish from a registered data-store location, e.g. from a file gdb in a registered folder or an sde connection registered as a database in the server data store.

Its just the way it works, server needs copies of the source documents in the directories it needs them in:

The domain arcgisserver system account handles all writes to the input folder when you publish or overwrite a service, not users. When you delete a map service, the service's input folder and subfolders should be deleted as well. If the input service name folder and subfolders do not delete when you delete a map, image or gp service, then you have or had other problems, probably related to sharing, where the arcgis server domain account could not fully access all the directories.

If you have an 'orphan' directory, make sure the reference is no longer in the portal content (if you are federated) before deleting it manually.

There should be no cleanup mode set on the input folder.

For the output folder, all files created by the arcgis server service account (via the REST endpoint are cleared out by the time set, usually 10 minutes. What I have found is files in the output directory that are not auto-cleaned are lyr files created by SOAP when someone at sometime connected to a service in ArcMap.

Hope this helps-

David Coley

Sarasota County GIS

Re: arcgisinput folder: What is it for?

HenryLindemann — Wed, 04 Aug 2021 13:58:39 GMT

So the only reason your arcgisinpput folder will get large is if you are copying the data and not referencing it when publishing, this happens when ArcGIS Server can not read the source data.

I would also not delete anything out of here as the content is referencing other configs in other locations.

What I would check is the following is the data sources registered in ArcGIS Server

if no then the service is copying the data.

so if your data is not on the server make sure you are using a domain service account for ArcGIS and that the folder is registered as a network share and that the service account has access to it

Re: arcgisinput folder: What is it for?

jcarlson — Wed, 04 Aug 2021 14:08:32 GMT

So should it be doing this for hosted services, then?

Re: arcgisinput folder: What is it for?

DavidColey — Thu, 05 Aug 2021 01:39:07 GMT

@HenryLindemann brings up a good point that I did not address in my reply. He is correct. I assumed that since you were referring to 'one and done' map services that you were publishing services by reference, and not copying data.

Re: arcgisinput folder: What is it for?

HenryLindemann — Thu, 05 Aug 2021 06:32:46 GMT

The Hosted services are stored in the ArcGIS datastore -a black box sql solution- the folder you see there is just a reference folder that stores details on how to access the data, these services should be deleted out of Portal otherwise you might create orphaned services, if you do create orphaned services you will have to delete them out of ArcGIS Server Manager.
Kind Regards
Henry

Re: arcgisinput folder: What is it for?

jcarlson — Thu, 05 Aug 2021 13:21:55 GMT

Ah yes, sorry about that. I simply meant that for many of our services, we don't actually make any changes to them once they are published, so I wasn't sure if the services would actually need the original source data.

Re: arcgisinput folder: What is it for?

jcarlson — Thu, 05 Aug 2021 13:25:25 GMT

That makes sense, and obviously following the "proper" procedure for deleting stuff is the way to go. I'm just trying to understand how this folder is actually functioning, though. If my hosted layer is in DataStore, is anything actually looking at or using these reference files/folders? It doesn't look like the files have been modified since their original publish date.

Re: arcgisinput folder: What is it for?

HenryLindemann — Thu, 05 Aug 2021 14:43:36 GMT

I hade a look at what is in there, from what I do on a day to day deploy, audit, recover , migrate.
I would need the data in these folders for example the v101 and p20 p14 folders contain the original mapx and mxd these always get lost, and the manifest provides information on where the data originated from the users etc.

and there might be some other features as well.

In my chase the folders are less than a 1 MB, but I see you have SD files and these can be large, in ArcGIS Enterprise 10.8.1 - 10.9 the sd files are moved to portal and then you can delete them but you can also use them to rebuild the service should it crash

if there are still folders after you removed a service that is not right, there might be a script problem if you use that I don't know how you remove services currently.

Kind Regards

Henry

Re: arcgisinput folder: What is it for?

DavidColey — Thu, 05 Aug 2021 20:01:21 GMT

Ok, so on a limb here - I think it's fine to know what's in the system folders for server, but they are system folders for a reason. The system account needs to manage these, not the administrator. In almost every case I can think of, going into these folders an manually deleting, updating, writing to or adding files is totally unadvisable.

Re: arcgisinput folder: What is it for?

jcarlson — Thu, 05 Aug 2021 20:10:11 GMT

I would agree with that, and I don't usually touch system files unless I'm 100% sure of what I'm doing. I only happened to notice it when I was clearing out some stuff that didn't delete when I removed a service.

As @HenryLindemann notes, it's good to hang onto SD files, but in the case of raster imagery, I've got a 10GB file, and unless I switch to keeping that file offline somewhere, I basically have to allocate 20GB of storage to publish the service, and that's before caching any tiles. But I guess that's just the way things are.

Another question, then: does DataStore not store raster imagery, just vector layers?

Re: arcgisinput folder: What is it for?

DavidColey — Wed, 11 Aug 2021 18:00:53 GMT

no it doesn't, not like a file gdb or an enterprise rdbms can store a blob or raster data type. Like when you create a tile cache for a hosted feature layer, that tile cache is going to reside in the cache directories, not in the relational datastore. At least that's my understanding....

For imagery, we use the image server role on our federated site. The source imagery resides in an enterprise file share, like a san, and the mosaic datasets that source the image services reside in a file gdb on the federated sites' data-store directory