Alaska ArcGIS Server SIG (AGS-SIG) – Dec 11, 2014 meeting summary

[RebeccaStrauch__GISP]

AGS-SIG 12/11/14 (online meeting) Presenter: Rebecca Strauch, GISP‌

Topic:  Working in the weeds with Alaska Geoportal Committee Data (AGC/SDMI imagery) - Tips and caching test results

Note/Tip: We are testing this site for meeting summaries vs. emailing slides or posting elsewhare. This summary was created as a “Discussion” which allows users that are logged in to “comment” but does not require an “answer”. Use your GeoNet username (originally called the “esri global account”) or you can create a new username/profile from link in upper-right corner, if needed.  Read-only mode should not require a login.

Highlights:

• Getting into the weeds, that is, SIG meeting will talk details. With transition of Governor and staff, AGC (Alaska Geoportal Committee, formerly SDMi-State Digital Mapping Initiative) may be paused until things settle down. Technical Advisory Group (TAG) and Technical Working Group (TWG) still working at macro level not into the weeds in discussion.
• Trying to get the discussion started on using the imagery and products that we will be getting from the AGC/SDMI process.  We now have enough data to start proof of concept testing, although still many holes. Data will continue to arrive over net year(s)
• Not necessarily a replacement for any of our current services or GINA BDL, but need to investigate data/tools to keep options flexible for staff (including remote/offline) and web maps/apps.
• Testing goals:
  • Determine new tile cache schema…projection, addition scale levels,
  • mosaic datasets,
  • Image Server extension,
  • ArcGIS Server 10.2.2 vs. 10.3 (pre-release)
• Step 1: reorganize folder/file structure so “like” images are grouped in folder.   Used the DOS command xcopy with ability to submit command that would use two wildcards/variables to iterate thru all the folders to pull files. See graphic below for sample restructure. [Note: too hard to show demo due to meeting display issues….will write up document explain process and post to this site…if needed asap, contact me direct]
• 
• Step 2: Watched several tech session and webinar videos, read online docs, discussed specific questions with esri staff.
  • UC2014 tech sessions, for those with UC recordings (~75 minutes each)
    • Caching Imagery Using ArcGIS
    • Managing Imagery and Raster data Using Mosaic Datasets
  • Other web resources
    • Shared Cached Imagery in ArcGIS (60-minutes ...gets into linking metadata at end)
    • Tutorial: Creating a cached image service
    • ArcGIS Resources: Image Management Workflows
    • ArcGIS Resources - community main page....includes links to other videos
    • Scripts and tools available thru Image Management Workflows
• - Step 3: Started with RGB/Imagery files
  • Tested many processes, but basically current steps are:
    • Create new FGDB
    • Create new Mosaic dataset – eventually, one for each Type of imagery .. Using projection “NAD83_CORS96_Epoch2003/Alaska_Albers” wkid #102247 Note: currently using Alaska Albers NAD83 wkid 3338. Parameters seem to be the same. Talked to Joel C (NPS) some about the differences.  Bottom line for most of my department’s data using 3338, we will just force the projection change, since there isn’t currently a transformation that would make a difference. Maybe we can get Joel Cusick  to post a doc or discussion.
    • Add raster files, as“workspace”, e.g. add the “RGB” folder for each year for the imagery. Allowed it to create overlays since it didn’t take long
    • Exported the “boundary” to a new feature class, to use for caching boundary
    • tested, created Imagery service directly from mosaic dataset
    • tested, created new tile schema, w/102247 and source scale 1:9449
      • 1:9449  is this really the source scale? Note: I read data into ArcMap and zoomed to source resolution.  If anyone has better info, let me know.
      • Later, added addition scales 16k, 32k and 5k to current “standard” tiling cache scales
        • I will start a different discussion re: the tile schema.  Also, will post preliminary config files.  We need to agree on a “standard” tile schema to facilitate the sharing of cache tiles (i.e., no need for all to cache their own).  Additional scales can always be added locally, if an aency needs additional levels.
      • tested, created cache in various ways and versions.  Note: I’m extremely impressed with the speed of caching using the mosaic dataset and Image Server extension (ISe) vs caching from a map service (MS). Was able to cache available RGB …not complete state, of course …with all the current scales, plus the source, 16k, and 32k scales in about 4 days vs estimated 6 weeks with a MS, even without those added scales….although I have not tested with current data, yet.  That is partially due to ISe caching at the source 1:9449 first..caching 1:16k from that cache, 1:24 from 1:16, etc.  from what I can understand, 1:5k would go back to the source, so processing that in separate process…current about 68 hrs into process, expect a couple more days. Bottom line…ISe caching has proved to be extremely fast…relative to map server caches anyway. 
      • tested “upgrading” 10.2.2 bundled cache to 10.3 bundled cache, but getting error…waiting to test in final
  • Plenty to still test
    • 10.3 (final) – new compact cache, compatible with 10.x? (prelim results...)
      • Desktop 10.0 – can see 10.2.x/10.3 as service, but not cache files in folder created by 10.3 (as bundled 10.3 format).  Can see 10.2.x folder
      • Desktop 10.2.2 – can see 10.2.x/10.3 as service, AND cache files in folder created by 10.3 (as bundled 10.3 format…or 10.2)
      • ....this may control the decision on whether to:
        • create in 10.3;
        • or whether we'll even “upgrade” from 10.x to 10.3 bundle, although if exported in 10.3 is supposed to be in 10.2 format
    • using fx (functions) of Image Server extensions, especially with DEM (dynamic caching?)
    • Tile packages (.tpk) using ArcMap as an alternative to caching for distribution, and for use with AGOL, etc
    • Getting the metadata connected…..
      • python to attach to xml to each tif?
    • Create service for the metadata only….look into steps from 60-minute video
    • figure out how to update with new data as it arrives
    • figure out if/how to fill in the holes in the data, at least temporarily
  • Found out that several folders did not contain all imagery types....not sure if these images are to-be-delivered due to cloud cover or processing issues, or just missing.
  • trying to determine how much of the "support docs" are duplicates....maybe can consolidate.
  • need to determine why metadata is missing for some imagery

Several questions were asked throughout the meeting:

- re: licensing and distribution of caches were asked…I[note: deferred answers to someone more knowledgeable from AGC/TAG]

-Jason asked about the Xcopy tools Becky used to restructure the files so she could more easily work with the imagery. [note: as mentioned above, I will create a separate writeup for the xcopy commands I used]

-Kim Homan is working with CBJ lidar, imagery, countours, hillshade etc….to build image services.  Ran into some issues:

1. State Plane projection required, output naming convention  has caused some issues… Silverlight doesn't accept reprojecting on the fly. If projections really are the same, then may be need to use alternative wkid??
2. Statistics - tile cache reporting tools become disconnected & reporting at 0%.  Becky deleted the status gdb and re-created to get this to work.  Also chatted about how the size estimator was also way off for Becky when she did her cache.

-Evan Thoms talked about his use of mosaic datasets and issues when trying to accommodate users on multi-versions of ArcGIS Desktop.  He is trying to move into image services with new Server deployment - wants to stand up options for web apps.  Created GDAL libraries (virtual rasters) with a few large raster  datasets (Digital Globe) - and works in both ArcGIS and QGIS.

     Becky asked Evan to present for the SIG ...tentatively Tues Jan 13, 2015.  May combine with update from this meeting and/or other topic (Evan mentioned his talk would be short)

Summary: Making progress, more to test.  Let’s keep in communication to collaborate efforts, testing, and eliminate duplication. Please add comments, especially if you have comments on items that i have tested or anything on the to-do list.

Thanks

Becky

Rebecca Strauch, GISP‌

Message was edited by: Rebecca Strauch updated. Added graphics and fixing training/tutorial links.

Message was edited by: Rebecca Strauch  Added Q&A notes from the meeting.

Message was edited by: Rebecca Strauch  --replaced title page graphic to make it easier to read.  no other changes