Community

Synchronize NetCDF files into Mosaic Dataset

1130
1
10-12-2017 06:07 PM
TilmannSteinmetz1
by
New Contributor II
‎10-12-2017 06:07 PM

Using either of ArcMap 10.5.1 and ArcGIS Pro 2.0.1 I am trying to keep a Mosaic Dataset updated with the contents of a folder of NetCDF files - where new NetCDF files are constantly being added --  but "Synchronize" doesn't seem to be doing what I expect as it fails to add new items:

  1. For my testing, I've created a new FileGeodatabase and a Mosaic Data set inside it.
  2. I've added a few NetCDF files to the MDS using the NetCDF raster type- works fine (using "Add rasters" Tool)
  3. I then move a few more NetCDF files into the directory where the first NetCDFs live
  4. I then run "Synchronize Mosaic Dataset" - hoping that both the first and the added NetCDF files would appear as references in my attribute table - but it's still showing  just the original ones, that were added using "Add rasters". When I run synchronize I use the "Update With New Items" Option. This is the command expression in Python - here with a SQL filter, but I've tried it with and without that with no difference:

    arcpy.management.SynchronizeMosaicDataset(r"D:\OneDrive - NIWA\GISDATA\ArcGISPro_Projects\NetCDF_Synchronizing_CSM\NetCDFs.gdb\syncWithNoFilterNoDatafileExtension", r"Name LIKE '%nwpsfc\_2017%' ESCAPE '\'", "UPDATE_WITH_NEW_ITEMS", "SYNC_STALE", "UPDATE_CELL_SIZES", "UPDATE_BOUNDARY", "NO_OVERVIEWS", "NO_PYRAMIDS", "NO_STATISTICS", "NO_THUMBNAILS", "NO_ITEM_CACHE", "REBUILD_RASTER", "UPDATE_FIELDS", "CenterX;CenterY;Dimensions;GroupName;ProductName;Raster;Shape;StdTime;Tag;Variable;ZOrder", "UPDATE_EXISTING_ITEMS", "IGNORE_BROKEN_ITEMS", "OVERWRITE_EXISTING_ITEMS", "NO_REFRESH_INFO", "NO_STATISTICS")

  5. I've tried a couple of different versions of the above process - using a SQL clause as mentioned by Jake Skinner in another GeoNet post here:

    https://community.esri.com/thread/121638

  6. As I'm using Windows 7 I've tried to disable Background Geoprocessing as advised here http://support.esri.com/en/bugs/nimbus/TklNMDYwNjAz - this had made no difference.

Is it the fact that I'm dealing with NetCDF files in the synchronize process that makes it fail to update the Mosaic Dataset? Anyone else had this? Sounds like the above mentioned old thread is about the same or a similar issue - but from the description of the synchronize tool it can be used to "add new items" which is exactly what I need to do on a regular basis - but it fails. Using the manual "Add rasters" tool instead doesn't sound like a valid option? mosaicdataset netcdf synchronize

‌

This is the output from one of my runs:

Start Time: Friday, 13 October 2017 1:26:17 p.m.

2017-10-13T13:26:18.275: Synchronizing items associated with raster type instance 'NetCDF' [ID: 1].

2017-10-13T13:26:18.297: Completed synchronization: 29 items selected, 0 items synchronized.

2017-10-13T13:26:18.308: Adding new data source items associated with the raster type instance 'NetCDF' [ID: 1] last used on 2017/10/13 13:19:18.127 (local).

2017-10-13T13:26:18.309: Loading raster datasets

2017-10-13T13:26:18.324: Completed crawling 0 data source items. Added 0 mosaic dataset items.

2017-10-13T13:26:18.326: Error: 8004205f: No new mosaic dataset item was added.

Edit:

Thinking that it is the NetCDF file format which doesn't synchronize properly:

The next output is after adding some other (non-NetCDF based) rasters to the same Mosaic Dataset and running a synchronize with some more of those afterwards - as you can see the NetCDF sync fails with "Error: 8004205f: No new mosaic dataset item was added." but synchronizing the other rasters is successful.

Executing: SynchronizeMosaicDataset "D:\OneDrive - NIWA\GISDATA\ArcGISPro_Projects\NetCDF_Synchronizing_CSM\NetCDFs.gdb\syncWithNoFilterNoDatafileExtension" # UPDATE_WITH_NEW_ITEMS SYNC_STALE UPDATE_CELL_SIZES UPDATE_BOUNDARY NO_OVERVIEWS NO_PYRAMIDS NO_STATISTICS NO_THUMBNAILS NO_ITEM_CACHE REBUILD_RASTER UPDATE_FIELDS # IGNORE_EXISTING_ITEMS IGNORE_BROKEN_ITEMS SKIP_EXISTING_ITEMS NO_REFRESH_INFO NO_STATISTICS

Start Time: Fri Oct 13 15:26:46 2017

2017-10-13T15:26:46.101: Adding new data source items associated with the raster type instance 'NetCDF' [ID: 1] last used on 2017/10/13 14:39:54.022 (local).

2017-10-13T15:26:46.102: Loading raster datasets

2017-10-13T15:26:46.115: Completed crawling 0 data source items. Added 0 mosaic dataset items.

2017-10-13T15:26:46.117: Error: 8004205f: No new mosaic dataset item was added.

2017-10-13T15:26:46.118: Adding new data source items associated with the raster type instance 'Raster Dataset' [ID: 2] last used on 2017/10/13 15:10:06.575 (local).

2017-10-13T15:26:46.120: Loading raster datasets

2017-10-13T15:26:46.149: Completed crawling 25 data source items. Added 25 mosaic dataset items.

2017-10-13T15:26:46.150: Synchronizing crawled data source items

2017-10-13T15:26:46.158: Synchronizing items associated with raster type instance 'Raster Dataset' [ID: 2].

2017-10-13T15:26:48.083: Completed synchronization: 25 items selected, 25 items synchronized.

2017-10-13T15:26:48.084: Processing duplicate mosaic dataset items

2017-10-13T15:26:48.105: Removed 13 duplicate mosaic dataset items.

2017-10-13T15:26:48.109: Computing cell size levels

2017-10-13T15:26:48.110: Computing unique cell size values

2017-10-13T15:26:48.189: Computing maximum cell size values

2017-10-13T15:26:48.253: Updating maximum cell size values

2017-10-13T15:26:48.261: Computing minimum cell size values

2017-10-13T15:26:48.381: Updating visibility values of selected items

2017-10-13T15:26:48.400: Updating visibility values of selected items

2017-10-13T15:26:48.406: Computing maximum cell size for mosaic dataset

2017-10-13T15:26:48.458: Completed computing cell size ranges.

2017-10-13T15:26:48.515: Completed building boundary.

Succeeded at Fri Oct 13 15:26:48 2017 (Elapsed Time: 2.49 seconds)

0 Kudos
1 Reply
SimonJackson
by
Occasional Contributor III
‎02-01-2019 01:42 AM

Having similar-ish issues.

  • Have a script that downloads NetCDF which has new time-slices of data in it (weekly forecast)
  • This overwrites the previous NetCDF in a folder
  • Mosaic Dataset points to this NetCDF file
  • Runs a Sync-MD 
  • Somehow still has a reference to the older forecast data, dispite removing overviews as part of the Sync. 

Any ideas?

arcpy.SynchronizeMosaicDataset_management(in_mosaic_dataset=r"😧\Work\BOM\workings\BOM_FTP.gdb\T_SFC_ArcMap", where_clause="", new_items="UPDATE_WITH_NEW_ITEMS", sync_only_stale="SYNC_STALE", update_cellsize_ranges="UPDATE_CELL_SIZES", update_boundary="UPDATE_BOUNDARY", update_overviews="UPDATE_OVERVIEWS", build_pyramids="NO_PYRAMIDS", calculate_statistics="CALCULATE_STATISTICS", build_thumbnails="NO_THUMBNAILS",build_item_cache="NO_ITEM_CACHE", rebuild_raster="REBUILD_RASTER", update_fields="UPDATE_FIELDS", fields_to_update="CenterX;CenterY;Dimensions;GroupName;ProductName;Raster;Shape;StdTime;Tag;Variable;ZOrder", existing_items="UPDATE_EXISTING_ITEMS", broken_items="REMOVE_BROKEN_ITEMS", skip_existing_items="SKIP_EXISTING_ITEMS", refresh_aggregate_info="NO_REFRESH_INFO", estimate_statistics="NO_STATISTICS") print('Finished syncing Mosaic Dataset') if __name__ =='__main__': main()

0 Kudos