topic File size increase when projecting raster in Imagery and Remote Sensing Questions

File size increase when projecting raster

forestknutsen1 — Tue, 06 Oct 2015 19:17:09 GMT

Why when is it that when I project a raster from Alaska state plane zone 4 feet to Web Mercator does the file size increase so dramatically? It starts at ~29MB before it is projected and then increases to ~257MB after projection.

NAD_1983_2011_StatePlane_Alaska_4_FIPS_5004_US_Survey_Ft

WGS_1984_Web_Mercator_Auxiliary_Sphere

The only thing I can think of is that the distortion in web mercator at high latitudes is causing the raster to be "stretched" in some way and then leading to an increase in file size.

Thanks for any input.

Re: File size increase when projecting raster

ChrisSmith7 — Tue, 06 Oct 2015 19:24:30 GMT

Esri has a technical article discussing some of these reasons:

28470 - Why does the data bit depth increase when a raster is projected, rotated, or clipped?

This can result in file size changes:

30560 - Why does the file size of my raster image change when I export or create a new image?

Re: File size increase when projecting raster

GabrielUpchurch1 — Tue, 06 Oct 2015 19:29:10 GMT

Interesting. It appears there has been a considerable addition of rows and columns. The proportional difference in rows and columns is almost identical to the proportional difference in file sizes. I would symbolize your NoData cells to see if that is where all the extra rows and columns are hiding.

Re: File size increase when projecting raster

ChrisSmith7 — Tue, 06 Oct 2015 19:29:36 GMT

Also, looking at your image, raster_cut_small contains more columns and rows; the cells are smaller, and there's a higher level for pyramids. All of this results in more data contained within the file.

Re: File size increase when projecting raster

forestknutsen1 — Tue, 06 Oct 2015 19:48:02 GMT

Thanks for the info; however, both images are 8 bit.

Re: File size increase when projecting raster

forestknutsen1 — Tue, 06 Oct 2015 19:56:35 GMT

Gabriel Upchurch:
Interesting. It appears there has been a considerable addition of rows and columns. The proportional difference in rows and columns is almost identical to the proportional difference in file sizes. I would symbolize your NoData cells to see if that is where all the extra rows and columns are hiding.

For sure the extra size has to be coming from the increase in rows and columns. But why is this happing? I would think that it would have more or less the same number of rows and columns as the original because:

Which is the cell size that the tool selected.

Re: File size increase when projecting raster

GabrielUpchurch1 — Tue, 06 Oct 2015 20:01:42 GMT

It appears you are confusing comments by two different people. You are exactly correct about the cell sizes since there was a conversion from feet to meters between the two systems. The cells are still the same size so this is not why the datasets are different sizes.

Did you symbolize the NoData as I (Gabriel Upchurch) suggested? My guess is that the dataset was rotated during the transformation which resulted in a large areas of NoData cells being added in the corners.

Re: File size increase when projecting raster

ChrisSmith7 — Tue, 06 Oct 2015 20:07:28 GMT

Yes, you're right - the reprojection to web mercator results in smaller cells from the conversion to meter, from foot. Sorry - should have clarified that as an observation rather than the cause.

Re: File size increase when projecting raster

GordonSumerling — Tue, 06 Oct 2015 20:21:10 GMT

Forest,

what hat compression is being applied to the output image? Is it the same as the source. Under the Environments there is a raster storage setting called compression which specifies the compression used on the image. See the help topic

Depending in in the source compression and the resultant image you could see these size changes.

cheers

Gordon

Re: File size increase when projecting raster

ChrisSmith7 — Tue, 06 Oct 2015 20:23:22 GMT

Also, in the past, I have run into problems with the raster environment settings - ArcGIS Help (10.2, 10.2.1, and 10.2.2)

Re: File size increase when projecting raster

forestknutsen1 — Tue, 06 Oct 2015 20:25:02 GMT

I am sorry that I mixed your comments up Gabriel and Chris.

Gabriel, the extra rows and columns are not in no data (well a few of them are for sure but not 1000s of them).

No data in magenta.

Re: File size increase when projecting raster

forestknutsen1 — Tue, 06 Oct 2015 20:26:24 GMT

Both are LZ77

Re: File size increase when projecting raster

ChrisSmith7 — Tue, 06 Oct 2015 20:37:07 GMT

For LZ77, there is no appreciable difference in file size between level 5 and 6 pyramids:

http://help.arcgis.com/EN/arcgisdesktop/10.0/help/index.html#//009t00000019000000

So that shouldn't be an issue. Actually, according to the documentation, the most this should cost you is 8%.

Re: File size increase when projecting raster

CodyBenkelman — Tue, 06 Oct 2015 20:43:57 GMT

Forest

There is something fundamentally wrong, but so far I can't figure out what it is. Most of the dialogue above is correct - it's not a difference in compression or units - but the number of rows and columns of your output should not be dramatically different than the input. I can't figure out why that would happen. It almost seems like you have a custom projection, and something in that projection definition is confusing the software - e.g. it automatically figures out that 0.75 feet = 0.228 meters, but then when reprojecting, it "forgets" that the units are changing, so it's outputting 9 identical pixels for every single input pixel (oversampling from 0.75 units to 0.228 units of the *same* units).

A few questions (although I do not believe that these explain the problem)

1) why are your rasters in a database? I typically recommend against that - rather, store as *.tif on disk and access them via a mosaic dataset. Is there a reason you can't do this?

2) Is there a reason that you MUST reproject and duplicate your data? If you use a mosaic dataset, you can project on the fly and not consume disk space or processing time...

3) have you reproduced this with other rasters, or is this one doi\ng something unique?

If you could send me your input file - or a smaller one that is doing the same thing - I'd like to try to reproduce it.

contact me for an upload link - cbenkelman@esri.com

Cody B

Re: File size increase when projecting raster

forestknutsen1 — Tue, 06 Oct 2015 21:31:29 GMT

1) I just put them into gdb for the projection processing as I have run into weird and unknown problems in the past when working with the tifs (this may still not be a problem. Last time I did this we still had a mixed environment 9.3 and 10.1. Now we have almost retired all of the 9.3 stuff)

2) The data must be projected for use in web maps. In the past we had a map service of a raster dataset that reprojected on the fly, but it was fuzzy when viewed at small scales. Now maybe a image service off of a mosaic dataset would not have this problem, but I understand that we would need an image server extension for ArcGIS server. The current plan is to have a cached map service on the hard drive of server in the same projection as the web mapping application.

3) This is a problem for all of our 2015 and 2012 ortho images.

Sadly, I can not share any part of the data as it is confidential; however, I will try and find something that I can share with you that has the same behavior.

Re: File size increase when projecting raster

CodyBenkelman — Tue, 06 Oct 2015 21:47:51 GMT

Forest

Got it -

[disclaimer - text here has been edited to correct a misstatement I made previously]

This may appear that Project Raster is in error but (as noted in my summary posted 10/12) it is actually honoring the definition of the Web Mercator projection which results in significant (but predictable) distortion at latitudes far from the equator.

One note, have you tried directly generating raster tile cache from the source data? I'm now concerned that you're duplicating data twice (state plane --> web mercator --> cache).

[another edit - raster tile cache is not suitable if the data is 32 bit (float) per pixel ]

If you're not aware, you can generate cache using ArcGIS Desktop. We did a recorded webinar (accessible at this link Esri Training | Sharing Cached Imagery in ArcGIS ) that explains some of the details.

Cody B.

Re: File size increase when projecting raster

forestknutsen1 — Tue, 06 Oct 2015 23:21:45 GMT

Great. We don't want two data sets if we don't need them. I will review the webinar.

Re: File size increase when projecting raster

GabrielUpchurch1 — Wed, 07 Oct 2015 12:26:39 GMT

Thanks for checking the NoData. I did some testing this morning and was able to reproduce your issue and think I have figured out what is occurring. It appears you are encountering the common issue of projection distortion. In the case of Web Mercator the distortion is largely in distance and area and with your data being so far from the system's origin, the distortion is sizable. To put it simply, a 0.228 meter cell in Web Mercator does not cover an equivalent area on the ground as a 0.75 feet cell in Alaska State Plane. To represent the same area on the ground while complying with the specified output cell size, the software is adding additional cells. The behavior is further exacerbated because you are going from a local coordinate system that is highly accurate in terms of distance and area to a global coordinate system that is highly inaccurate in terms of distance and area.

The additional cells are not obvious because the software does a good job of maintaining the pattern in the data. I didn't have any AK data so I generated a random raster for testing purposes and I was unable to visually determine that cells were added. To confirm that cells were actually being added during the projection process I used the Raster to Point tool.

This first screenshot shows points representing the cell centers in the Alaska State Plane dataset.

This second screenshot shows points representing the cell centers after projecting to Web Mercator.

Re: File size increase when projecting raster

ChrisSmith7 — Wed, 07 Oct 2015 12:58:41 GMT

I wondered the same when Forest created the thread - he even mentioned that as a possibility. Alaska would definitely experience a high degree of distortion in web mercator - thanks for providing examples.

Re: File size increase when projecting raster

forestknutsen1 — Wed, 07 Oct 2015 16:13:00 GMT

It does feel like this is the only possibility, other then the tool just having some error that is expressing it's self in this situation. But I would think that the tool would compensate for the distortion as one moved to different areas of the projection (kinda the point of projecting). I can manually set the cell size larger, like .4m and produce cells that are still smaller then .75ft when displayed in Arc, and not have loss in image quality.

I sent a sanitized raster to Coby, the esri person on this thread, for testing. So, we will see what they say...