How long should projecting an orthophoto take?

[MarkHotz]

I created a mosaic from 411 TIFF orthophotos.   The TIFFs total 106 GB in size and each are in UTM. However, the final image needs to be projected for our web map (Portal) as it's typically not wise to allow it to reproject on-the-fly over a LAN when you want users to have a relatively fast and responsive mapping experience; hence, it should be in the same projection used by Portal.

So I'm in process of projecting the Mosaic using the raster projection tool.  I opted to run this through Modelbuilder so I can at least see if the process is indeed processing.  However, actual progress, I find, can only be done by monitoring the size of the folder where the final image is going (using Windows Explorer).

The problem I'm going to have is time.  This folder into which the projected Mosaic will be deposited is growing at about 18 mb/minute, and the resulting file won't be 106 GB...it will more than likely end up being around 150 GB in size.  By my calculations (and correct me if I'm wrong), the resulting file is growing at approximately 1 GB/hour; hence, at this rate the projected image will be completed in approximately 150 hours....This means at this rate the final image should be completed in about 6 days.  Cripes.

Does this should right to you?  Should a simple projection of an image take almost a week to process in ArcMap?  Is there perhaps a faster way to do this?

Thanks

[JayantaPoddar]

Maybe a standalone python script would do it much faster.
Project Raster—Help | ArcGIS Desktop 

[MarkHotz]

Jayanta:

Thanks for the response.  I think using this might help track it's progress better, but it would be using the same geoprocessing tool (raster projection) so I'm surmising that the process would take just as long.  However, it could allow it to process separately in ArcGIS Pro...I should try that next.

Thanks.

[RebeccaStrauch__GISP]

I know you said you don't wan't to reproject on the fly, but you mention that you have a mosaic.  Have you even tried to see if the reproject function will work for you?  It does reproject on the fly, but only the extent and scale that is viewed, from what I understand (still testing myself).  It's worth a try if you haven't already tried it.

Reproject function—Help | ArcGIS Desktop 

Other help topics leading up to that page...

- What are the functions used by a raster or mosaic dataset?—Help | ArcGIS Desktop 

- Image Management 

[MarkHotz]

Rebecca:

Part of the problem I am having is that when I try to publish this ortho to a map service to use in Portal I'm presented with error messages that tell me my licensing won't permit it.  So I'm surmising that in order to publish a mosaic to a map service then one requires the image server license, which we don't have, nor could we ever justify purchasing.

I have successfully masked, clipped and copied mosaics to a file geodatabase in the past, but this then loses all mosaic functionality and creates a flat image file (a composite file).  I have also found that building pyramids and using any compression during the process tends to degrade image quality and does not look good.  However, a raw file is OK for our purposes anyway as we're only using this ortho as an up-to-date base map, and once cached on ArcGIS Server it works really well.  In the GIS itself a raw flat file, although a little slower to draw, still works well, and there are other caching processes to be explored here as well; however, the mosaic can still function in this environment too anyway.

As for projection on-the-fly, both ArcGIS and Portal will perform this function automatically, which is fine for small data, but for large ones like ortho or satellite imagery it is not really recommended as this will only slow down rendering on screen, and users will tend to be reluctant to use it if latency is even 5 seconds or more.  I have created a map service with a raw composite orthophoto in Portal in the past (to experiment) and it takes 45 seconds to draw and refresh; hence, caching is really the best way to eliminate this issue.  Once cached the ortho image will draw to the screen in Portal in less than 1-second, but having everything in Web Mercator in Portal is probably a good habit to get into.

This all then reverts to the issue of how long should it really take to project a 150 GB orthophoto :-).  I think anything more than 4 hours is ridiculous, but that is just me LOL.  However, what I might try next is to reproject this same ortho using SAFE FME to see if that works better.

[RebeccaStrauch__GISP]

Hi Mark,

Sorry, read your OP too fast and didn't piece together that you were trying to make a service (and didn't have Image Server).  For fwiw, I went thru the process of justifying Image Server (then an extension) a few years ago.  Now (10.5.x)  it is a separate product, and I am trying to upgrade from our current 10.2.x production).  We will not be federating Portal for the foreseeable future, so that is not in our workflow.

What sold me on the extension was cutting my caching of 10 scale from 8-10 days to 1.5 days, while adding 4 more detailed scales. [note: with Image Server, you first cached the source scale, then do the others....it works of the source scale cache.] To me that was worth the cost. 

But now, I'm looking more at the functions and mostly dynamic services (base map will still have a cache option).  We have about 800 GB of rgb imagery that I am currently working on (and it is not the standard WMA projection). 

I realize this may not help you any (Image Server isn't cheap) but if you are trying to justify it at all, it might.  Good luck with your projection efforts.

[MarkHotz]

Rebecca:

Thanks again for the response.  What this tells me is that I'm not alone dealing with this...misery loves company I guess LOL.  I cached two items last year:  One orthophoto, which took about 2 hours to cache, and one base map for a large region that actually took 4½ weeks to process.  Even though I managed to cache this base map, and it took a very long time, it will not be necessary to repeat this process any time soon; hence, Image Server simply cannot be justified.  This base map was a vector file but it just took far too long to draw in Portal so I managed to cache the map service, and now this draws to the screen in Portal in less than a second, so it certainly does help.

The only reason I cached this base map at all is because our Portal relies solely on ESRI (or Bing) satellite and base map imagery for base maps, and during a crisis if the web goes down (which it does frequently here) we would then be blind.  At the time the organization didn't want to pay $20,000 for a hi-res satellite image (although I should revisit that request again soon), so the least cost-effective method was to simply cache what takes too long to render.  At least it works.  I'm just glad that no one decided to reboot my computer when I wasn't around during the process :-).

[RebeccaStrauch__GISP]

....watch that you don't run out of disk space.  Something we don't necessarily think about anymore with disk space being fairly cheap these day. We still have older machine where expanding disk space is not easy, but they are also not used heavily, so good workhorse for doing things that take days/weeks.  But running a "create overlay" on my rgb mosaic last night brought it to 0 (at about 76% complete).  That was my 10.2.x machine as a comparison to an issue I'm having with my 10.5.1 test box. Oh the joys of having things run for hours/days/weeks and not finishing.

[MarkHotz]

Rebecca:

That's amusing...I have had a few "Homer Simpson moments" where right at the very end of a long process it crashes due to something...most often disk space :-).  I'm not trying to marginalize your experience here, but it is nice seeing that others are experiencing the same issues as me LOL.

The most recent disk space issue for me was when I was caching a map service.  We have 1 TB volumes for all of our add-on drives to make it easier for back ups (this decision was made long before GIS arrived here), but what I found when caching is that the ArcGIS estimated required disk space and the resulting disk space used are rarely the same (i.e. the result is always about ⅓ less than the estimated).  However, now I realize that the "estimated disk space" for caching has little to do with the final space required, but everything to do with how much space is required to actually process the cache, so if the estimated disk space required is 900 GB then one would probably need that much space just to process the cache, and the final result would be around 300 GB.  Thankfully, we're not caching a lot of imagery, or I'd be adding dozens of volumes every year :-).

[PaulDavidson1]

That time seems "normal."

We take about a week to process new flyover data ~ every two years.

Jayanta could be right though that Python would speed it up.

ModelBuilder seems to inject a lot of overhead.

I've seen one big MB tool go from over 60 min to about 6 min when moved into Python.

But that had a lot of calls across modules.

But, like you say, it's really the one geoproc call for you so...

But 64bit in ArcPro might make a difference.

Regarding the Imagery Server, have you considered an ELA?

Esri offers a Small Gov ELA.  If you can afford it, an ELA is the way to go in my mind.

You can quit worrying over licensing (except for named users.)

For us, it's money really well spent.  As a utility, we're blind without GIS.