Where to store imagery for best performance: in SDE?

[Anonymous User]

Hi all.  I did a bit of readingto find out how to get the best speed and quality for aerial imagery. We have 3" aerials that arrived as geoTIFFs.  I kept them in geoTIFF with 80% JPG compression in YCBCR color space to build pyramids and overviews.  Apparently JPG is much faster than lossless or wavelet (ECW/JPG2k) compression to serve per these links:

http://blog.cleverelephant.ca/2015/02/geotiff-compression-for-dummies.html

https://blogs.esri.com/esri/arcgis/2010/12/21/rasters-get-speed-save-space/

Additionally, bilinear convolution was used since imagery is continuous data. (i.e. not nearest neighbor as used for discrete data)

http://gis.stackexchange.com/questions/17328/what-resampling-technique-should-be-used-when-projectin... 

Now, my main question is this: I am inquiring where to store this in order to publish a service that is as fast as possible.  The SDE or a file geodatabase?  I intend to consume the imagery as a tiled service in web viewers to get the fastest perfromance. We have server 10.2.2. Eventually this year we'll migrate to 10.4. 

This link below states imagery referenced in a mosaic dataset (which is what should be used) should NOT be stored in a database, out of performance reasons, but as standalone files in the file system, and it also states overviews should not be stored in SDE either:

https://community.esri.com/docs/DOC-8061  (it's down a few pages, it's one of the responses from Esri)

In other words, if I understand correctly, the original image files themselves and the overviews should NOT be copied into a geodatabase. The mosaic dataset should point to the plain original files. What is not clear however is whether the mosaic dataset should be on the SDE or not, before publishing to a service.  Again, I think I've got the quality and size optimal from the parameterization above but I just want to know now where to store the imagery for max. speed.

Also, I noticed discussion of projection but that Esri recommends keeping imagery in its original projection. Our imagery was delivered to us in Georgia State Plane Ft NAD83. (no mention of which NAD realization in the metadata).

https://blogs.esri.com/esri/apl/2014/06/24/designing-and-optimizing-image-services-for-high-performa...

However based on this link above, I am wondering if for faster performance, if we stored imagery in web Mercator (reprojected) it would serve up approx. a second faster?  Because of course otherwise ArcGIS Server reprojects on the fly from State Plane to Web Mercator in our webviewers, all of which have Esri basemaps.   I am wondering whether forcing webviewers to State plane by loading our vector layers first would make Esri’s servers broadcast the Esri basemaps to the viewer in State plane? By loading our layers first to set the projection? Or if there's another way to get around this programmatically and get request Esri basemaps in a State plane projection vs web mercator?  On this last point I shall research more.  For now I shall leave it in its original projection. At least it's in mercator so the performance penalty should be lesser, according this document above.

[Anonymous User]

Cody Benkelman​ thank you very much for the help. Yes we have Image Server extension.

I have the mosaic stored in the way you suggest, with the imagery as plain files in the filesystem pointed to from a mosaic dataset in an FGBD. Imagery and mosaic are state plane. I was about to publish and wanted to confirm. I'd created pyramids and overviews.  I'd thought the overviews were for the web viewer tiles but I guess not and this cache process is the third step needed as you indicated.

Imagery in Web applications: Should I use a cached map service or an image service? | ArcGIS Blog

Imagery in Web applications: Should I use a cached map service or an image service? | ArcGIS Blog

Based on the above and your very helpful replies ... While I have never gone through this whole imagery publishing process before (can you tell!) our organization has historically used a custom tiling scale scheme: Imagery/Imagery_2013 (MapServer)

However it doesn't zoom enough, I want to see the 1:141 and 1:70.  So I also added two zoom levels by basing it off of the Esri World topo basemap service, per this thread:

Local cached services not displaying below a scale of 1128 in AGOL web map. 

The data frame for my imagery that I am about to publish as below is in Web Mercator. (The original imagery is State Plane and so the mosaic is also in state plane.) After reviewing your post Cody, I think it will make more sense to bring our imagery into the JS API web viewers and re-project it to fit into web mercator, versus reprojecting the Esri basemaps... because we have the Esri topo basemap on as the viewer basemap.  (We are a Community Basemaps city).  Users have to explicitly turn imagery.  So definitely a minority of the users will even use imagery. Thus Esri basemap speed while zooming/panning is takes priority over imagery, and I guess it is a choice of one over another. I will test as you recommend and see.

Currently I am waiting for imagery to copy. I am copying it in to the C:\arcgisserver\arcgiscache because the server manager didn't like the other network folder I had the imagery in when it prompted me to register the location, so I hope if I create an mxd referencing C:\arcgisserver\arcgiscache it will already be registered. Hopefully it will copy by the end of today so I can initiate this caching/publishing process. 

[Anonymous User]

I created pyramids and overviews for a subset of my imagery, for about 15% of the county and it seems to have worked. Curiously, first I created it with 90% JPG YBCR compression and it resulted in pyramids but with no compression. It says 90% JPG YBCR in the Environment and in Results in the geoprocessing it says it.  Weird. I then did the same operation but without compression. it also worked. Of course, no compression.

I was wondering though.... Should we use compression in pyramids?   Or only tiles?  Because I thought tiles used pyramids, in other words, is it compressing the imagery twice? 80% on pyramids and then on top of that another 80% compression? Is that correct conceptually? Or do the tiles come from the original files? (in which case I'd imagine you'd want compression for both)

Also... It seems clear I should go to publish the service, and when doing so, select tile map cache manually and then use the scheme from Esri World Topo Map service to dial down to 1:70. But question... can I first tile down to say 1:1100 then at a later point add tiles down to 1:70?  The first 20 tiles take a day the next levels will probably take weeks or months, so I was wondering if I can do the tiling in stages. Plus, to start using immediately, i'll publish as a dynamic map service also.

And, how about for Desktop? Is it faster to copy the mosaic dataset into SDE and have users add that way? Or have them add an ArcGIS Server service and pull the tiled map service? Note that Arc users are all on a fast LAN on the same subnet. Not sure if server is smart enough to keep the tiled map service i/o in the network.. I guess so? I guess that's a network/DNS question. Regardless I just want whichever is faster especially if the quality will be the same.  Note that although I'd add the mosaic to the SDE it's still in a file GDB pointing the native source files in the regular filesystem. And yes it's on the same machine as the Server and SDE.

I'll this thread updated because frankly it seems like the procedures I've outlined here are probably the solution for fastest/best aerial imagery storage and serving, for many (perhaps not all but the vast majority) of organizations serving imagery to web viewers and to Arc Desktop.  (Sure you may want to store in the SDE for granular security, or use some weird compression for some specific reason but the workflow I have devised seems like a most common use case)

[Anonymous User]

Apparently faster to have our data reproject to web mercator and use Esri basemaps, vs make Esri basemaps reproject to match State Plane.  It turned out most of our viewers are web merc since that was what loaded first and I didn't override it. However, even better, longterm, is I'll probably automate replicating our vector layers to a web mercator fGDB. So it's all web mercator. But I noted in the Esri performance whitepaper that this isn't as a big a deal as other bottlenecks as the datum is the same and projection is similar anyhow. Like tile vs dynamic.  I think it was Software Performance - GIS Wiki | The GIS Encyclopedia  somewhere or something similar that showed actual tests on throughput. I think an Esri blog. I'll try to track it down but I distinctly remember the graphs showing this wasn't the biggest bottleneck. Biggest for us was serving out of SDE, not having enough pools, and not having the best quality settings on tiled services and pyramids/overviews.

Here are my steps, this is kind of a note to self and I'll update once services are published in final version.

1. Copied imagery from vendor deliverable hard drive.
2. Added to mosaic dataset on C drive of service server machine  (and separately to other machine that runs SDE for desktop users)
3. Built pyramids  JPG YCbCr compression at 80 with bilinear convolution  (see 

   http://blog.cleverelephant.ca/2015/02/geotiff-compression-for-dummies.html for actual testing to show why to use this) 

4. Built overviews
5. Tiled down to 1:71, same as our imagery, TIFF with JPG 80 compression
6. Created tiled services in web Mercator projection (for use in web viewers)

Success. Works awesome. I will have to take a look more at the tile size as mentioned above. For now I left it at the 256x256 default Compact.  

Also I was a little confused, because when creating a tile cache for a service it never asks about resampling method.  I hope it's doing bilinear??? Let's just say, I have noticed imagery done with Nearest and it looks awful, as it should, as it's not continuous data and not getting smoothed.  So..  Surely the tiles are getting resampled to get created. I don't see where that setting is though. Also I made it go down to 1:70 but I would think it would be good to let users zoom even farther in. Once I get this service published again now that I'm circling back I'll check whether I can force dynamic rendering after the maximum LOD. ArcGIS Server: allowing tile resampling.  

Also I guess you can either add the mosaic dataset, and publish a service and cache that, or you can directly right click in Catalog and click "Generate Tile Cache".  Imagery in Web applications: Should I use a cached map service or an image service? | ArcGIS Blog  First I'm going to try directly as cached image service. Seems more logical.

There should be a clear Blog on creating aerial imagery services in one spot.  If necessary I'll continue to refine my work and findings and cite all appropriate literature and credit everyone to that end. This is a common workflow:  local governments get aerials.  We simply need a fast, good-quality aerial image service. No frills.  While we have DEMs and DSMs that's about it, all our imagery is either aerial or elevation both of which these concepts apply to as continuous data.  

It appears all the defaults must be changed for aerials. Nearest neighbor resampling is default. It says use bilinear for continuous but many analysts may not realize aerials are continuous. Recommendation: clarify this tooltip to say 'e.g., imagery. Another example, JPG YCbCr should be default. That is hidden away in Environment and a beginner analyst wouldn't know what that is let alone to check it. Must be done if publishing it as a map service mxd and you want superior jpg quality YCbCr. Also many agencies are still under the impression everything must be in SDE and Esri could publish more articles on making high performance services. And then last, I'll be scripting to export vectors to a web mercator fGBD to go over this web mercator image tiled service I'm creating for our latest ortho so everything is web mercator. And then for measurement tools I'll ensure they using geodesic calculation. 

Literature:

https://geonet.esri.com/docs/DOC-8061

http://desktop.arcgis.com/en/arcmap/10.3/manage-data/raster-and-images/improving-the-display-of-rast...

http://gis.stackexchange.com/questions/17328/what-resampling-technique-should-be-used-when-projectin...

http://blog.cleverelephant.ca/2015/02/geotiff-compression-for-dummies.html

https://blogs.esri.com/esri/arcgis/2010/12/21/rasters-get-speed-save-space/

https://blogs.esri.com/esri/arcgis/2012/11/14/should-i-build-pyramids-or-overviews/

 https://blogs.esri.com/esri/apl/2014/06/24/designing-and-optimizing-image-services-for-high-performa...