First of all, we have 5 data sets. The characteristics of these data sets are as follows:
- Data Set (A) => 72 files [NetCDF][~ 10 MB x 72 = ~ 720 MB]
- Data Set (B) => 65 files [GRIB][~ 3 MB x 65 = ~ 195 MB]
- Data Set (C) => 1 files [NetCDF][~ 920 MB x 1 = ~ 920 MB]
- Data Set (D) => 73 files [NetCDF][~ 7 MB x 73 = ~ 511 MB]
- Data Set (E) => 1 files [NetCDF][~ 2048 MB x 1 = ~ 2048 MB]
We wrote a python script to run at different times for each dataset. Our script takes 4 parameters as input. These;
1. Workspace Path (String) => "C:\MY\WrkSpc" [Directory where our Mosaic Datasets are located]
2. Spatial Reference (String) => "WGS 1984"
3. Root Directory Path (String) => "C:\MY\Data" [Main directory where our data sets are located]
4. Text File Name (String) => "Test.txt" [Name of text file on root path]
The main target of this script which contains the codes below is to read the text file on root directory, to get the file path list from text file, and then add the files to the respective mosaic dataset.
import sys, os, arcpy
arcpy_workspace = arcpy.GetParameterAsText(0)
spatial_reference = arcpy.GetParameter(1)
root_path = arcpy.GetParameter(2)
text_file_name = arcpy.GetParameter(3)
arcpy.env.workspace = arcpy_workspace
gcs_wgs_1984 = arcpy.SpatialReference(spatial_reference)
file_list = 
file_type = ""
mosaic_dataset_path_list = 
### Read text file ###
output_file = open(os.path.join(root_path, text_file_name), 'r')
### Get file list ###
file_list = output_file.readlines().split(';')
### ...DO SOMETHING...###
### file_type = "NetCDF" OR file_type = "GRIB"
### mosaic_dataset_path = os.path.join(arcpy_workspace, mosaic_dataset_name)
### and then add rasters
### Add rasters to mosaic dataset ###
arcpy.AddRastersToMosaicDataset_management(mosaic_dataset_path, file_type, file_list)
In fact, since we published the script as a geoprocessing service (GS), the script worked succesfully in all conditions. However, one day when the GS is running, server (Windows Virtual Machine) is shutting down and restarting after about 5 minutes. Here are some of the things that happen after this: [NOTE: First of all, I should say that when we run the script on ArcGIS Desktop 10.4.1, there are no errors that I will talk about below.]
When we want to add the files in A, B and D data sets containing multiple files to the mosaic datasets using GS, two different situations arise.
1.1 When we give the files as input to the AddRastersToMosaicDataset_managementfunction in a list, it returns succeeded, but does not add anything to MD, even though it does not return any error.
1.2 When we give the files as input to AddRastersToMosaicDataset_management function one by one, it returns succeeded and performs the addition process successfully.
When we want to add the files in C and E data sets containing single and large size files to mosaic datasets using GS, we see errors like this, "Failed to open raster item: [C:\MY\WrkSpc\C.gdb\AMD_C_CAT 10]". But GS is hanging here and waiting for the timeout period. When the timeout expires, it returns failed.
The solutions we tried were:
- Comparison of directory permissions for "Administrator" and "ArcGIS" users (both users are fully authorized in the relevant directories)
- Running GS service on another server (tried on 5 different servers but only worked on one)
- Running GS service on another server with ArcGIS server 10.5.1 installed
- Publishing a new GS service using the "AddRastersToMosaicDataset" toolbox (The same errors are still being received in the same situation)
- as much as possible to increase the heap size, upload file size, instance, timeout value
Thank you for reading. I look forward to your help or suggestions.