nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

File size increase when projecting raster

forestknutsen1

Why when is it that when I project a raster from Alaska state plane zone 4 feet to Web Mercator does the file size increase so dramatically? It starts at ~29MB before it is projected and then increases to ~257MB after projection.

NAD_1983_2011_StatePlane_Alaska_4_FIPS_5004_US_Survey_Ft

WGS_1984_Web_Mercator_Auxiliary_Sphere

The only thing I can think of is that the distortion in web mercator at high latitudes is causing the raster to be "stretched" in some way and then leading to an increase in file size.

Thanks for any input.

Vind meer berichten getagd met

Accepted answers

Cody Benkelman

Forest

I thought I should summarize, although the post by Gabriel Upchurch summarized this pretty well – and your initial assumption [that this is caused by the inherent distortion of Web Mercator at high latitudes] was correct. I should have realized the details sooner.

In the Web Mercator projection, the scale increases as (1/(cosine of Latitude)) so at 60deg it is 2x larger (2x scale error), and your data is at about 70 degrees latitude, so the scale error is almost exactly a factor of 3. Cosine(70) = .3420. That means it is a proper implementation for a Web Mercator projection that a distance of 100 meters will be reported to appear to be 300 meters. I had not realized this distortion would be embedded into a reprojected file, but it is necessary for compatibility with the coordinate system. Multiplied in 2 dimensions, this (3x3) distortio is a factor of ~9, so your input file (29 MB) becomes almost 250 MB after reprojection.

This really emphasizes the disadvantage of using a projected coordinate system too far from its origin. If all of your work will be at high latitudes, it would be better to use a local coordinate system, assuming any measurements will be made on the map.

Note I did some testing, and verified that the ArcGIS Online web map compensates for the distortion, so it provides generally accurate measurements. (Exactly how accurate, I do not know – I would need to ask our experts…)

Also note that ArcGIS Desktop has options for how it reports measurements. It will default to "PLANAR" measurements when using the measure tool (which does NOT compensate for the scale distortion), but the tool has the option to choose "GEODESIC" which will provide accurate local measurements.

Options to consider:

The best option is to use the Mosaic Dataset to serve your imagery directly from its original format, and the application that consumes it can request any projection (AK State Plane, Web Mercator, other) and ArcGIS will project on the fly. For this, you will need our Image Extension, and I’m trying to confirm if your client already has it. The advantage is you won’t have to duplicate data on disk (either as export to Web Mercator *or* in raster tile cache format).
1. Sub-option here – you do not need the image extension to serve one single image e.g. an orthomosaic of a single project site – but I’m guessing you have more than one site/one image and you would want to use a Mosaic Dataset & thus the Image Extension…
If the web map needs to support measurements, consider building your web maps/apps in Alaska State Plane or another appropriate coordinate system. It is not mandatory for web maps/apps to use the Web Mercator projection, but that raises the question of what other data must appear in your maps…
If you want to proceed with services using Web Mercator, I think they can be fine for visualization at Latitude 70 degrees (but at even more extreme latitudes I think the distortion would be unacceptable). If your users expect to make any measurements, you will need to consider and test the Web Mercator scale issues carefully.
If you do proceed in Web Mercator, I’d suggest you cache directly from the Mosaic Dataset. The cache may increase in size due to the scaling, but you should use JPG format (assuming your images are continuous imagery or a hillshade of elevation data), and keep the scale changes due to the projection in mind – if you want the visual quality of a level 19 cache, you can actually start at level 18 (1/4 the data volume) and you won’t lose visual quality. Again, we should test a small area first to be sure!
1. Note I had forgotten that your data file was 32 bit float. If this is elevation data, then the raster tile cache format is not usable – it is limited to 8 bits, 3 bands. We can share elevation data as a compressed 3D surface (different cache format), so if you need to share elevation values, this is a separate discussion. But note that using a dynamic image service, shared from a Mosaic Dataset, solves this…

All comments

Cody Benkelman

Leo

Web AppBuilder does not publish, so I don't understand what you are trying to do. and no, it also does not reproject. However if you have an image service published (through image server) from a mosaic dataset, web appbuilder can request any (known) projection, and the server will reproject on the fly.

Can you start a new thread and explain more completely what you have, and what you're trying to do? Also let us know what version of ArcGIS Desktop (ArcMap? ArcGIS Pro?) you are using

Cody B

LeoLadefian5

I'm having trouble with publishing directly from mosaic (state plane) using webappbuilder. I guess webappbuilder doesn't reproject on the fly?

Cody Benkelman

Forest

Options to consider:

The best option is to use the Mosaic Dataset to serve your imagery directly from its original format, and the application that consumes it can request any projection (AK State Plane, Web Mercator, other) and ArcGIS will project on the fly. For this, you will need our Image Extension, and I’m trying to confirm if your client already has it. The advantage is you won’t have to duplicate data on disk (either as export to Web Mercator *or* in raster tile cache format).
1. Sub-option here – you do not need the image extension to serve one single image e.g. an orthomosaic of a single project site – but I’m guessing you have more than one site/one image and you would want to use a Mosaic Dataset & thus the Image Extension…
If the web map needs to support measurements, consider building your web maps/apps in Alaska State Plane or another appropriate coordinate system. It is not mandatory for web maps/apps to use the Web Mercator projection, but that raises the question of what other data must appear in your maps…
If you want to proceed with services using Web Mercator, I think they can be fine for visualization at Latitude 70 degrees (but at even more extreme latitudes I think the distortion would be unacceptable). If your users expect to make any measurements, you will need to consider and test the Web Mercator scale issues carefully.
If you do proceed in Web Mercator, I’d suggest you cache directly from the Mosaic Dataset. The cache may increase in size due to the scaling, but you should use JPG format (assuming your images are continuous imagery or a hillshade of elevation data), and keep the scale changes due to the projection in mind – if you want the visual quality of a level 19 cache, you can actually start at level 18 (1/4 the data volume) and you won’t lose visual quality. Again, we should test a small area first to be sure!
1. Note I had forgotten that your data file was 32 bit float. If this is elevation data, then the raster tile cache format is not usable – it is limited to 8 bits, 3 bands. We can share elevation data as a compressed 3D surface (different cache format), so if you need to share elevation values, this is a separate discussion. But note that using a dynamic image service, shared from a Mosaic Dataset, solves this…

forestknutsen1

Great. We don't want two data sets if we don't need them. I will review the webinar.

Cody Benkelman

Forest

Got it -

[disclaimer - text here has been edited to correct a misstatement I made previously]

This may appear that Project Raster is in error but (as noted in my summary posted 10/12) it is actually honoring the definition of the Web Mercator projection which results in significant (but predictable) distortion at latitudes far from the equator.

One note, have you tried directly generating raster tile cache from the source data? I'm now concerned that you're duplicating data twice (state plane --> web mercator --> cache).

[another edit - raster tile cache is not suitable if the data is 32 bit (float) per pixel ]

If you're not aware, you can generate cache using ArcGIS Desktop. We did a recorded webinar (accessible at this link Esri Training | Sharing Cached Imagery in ArcGIS ) that explains some of the details.

Cody B.

forestknutsen1

1) I just put them into gdb for the projection processing as I have run into weird and unknown problems in the past when working with the tifs (this may still not be a problem. Last time I did this we still had a mixed environment 9.3 and 10.1. Now we have almost retired all of the 9.3 stuff)

2) The data must be projected for use in web maps. In the past we had a map service of a raster dataset that reprojected on the fly, but it was fuzzy when viewed at small scales. Now maybe a image service off of a mosaic dataset would not have this problem, but I understand that we would need an image server extension for ArcGIS server. The current plan is to have a cached map service on the hard drive of server in the same projection as the web mapping application.

3) This is a problem for all of our 2015 and 2012 ortho images.

Sadly, I can not share any part of the data as it is confidential; however, I will try and find something that I can share with you that has the same behavior.

Cody Benkelman

Forest

There is something fundamentally wrong, but so far I can't figure out what it is. Most of the dialogue above is correct - it's not a difference in compression or units - but the number of rows and columns of your output should not be dramatically different than the input. I can't figure out why that would happen. It almost seems like you have a custom projection, and something in that projection definition is confusing the software - e.g. it automatically figures out that 0.75 feet = 0.228 meters, but then when reprojecting, it "forgets" that the units are changing, so it's outputting 9 identical pixels for every single input pixel (oversampling from 0.75 units to 0.228 units of the *same* units).

A few questions (although I do not believe that these explain the problem)

1) why are your rasters in a database? I typically recommend against that - rather, store as *.tif on disk and access them via a mosaic dataset. Is there a reason you can't do this?

2) Is there a reason that you MUST reproject and duplicate your data? If you use a mosaic dataset, you can project on the fly and not consume disk space or processing time...

3) have you reproduced this with other rasters, or is this one doi\ng something unique?

If you could send me your input file - or a smaller one that is doing the same thing - I'd like to try to reproduce it.

contact me for an upload link - cbenkelman@esri.com

Cody B

ChrisSmith7

For LZ77, there is no appreciable difference in file size between level 5 and 6 pyramids:

http://help.arcgis.com/EN/arcgisdesktop/10.0/help/index.html#//009t00000019000000

So that shouldn't be an issue. Actually, according to the documentation, the most this should cost you is 8%.

forestknutsen1

Both are LZ77

forestknutsen1

I am sorry that I mixed your comments up Gabriel and Chris.

Gabriel, the extra rows and columns are not in no data (well a few of them are for sure but not 1000s of them).

No data in magenta.

ChrisSmith7

Also, in the past, I have run into problems with the raster environment settings - ArcGIS Help (10.2, 10.2.1, and 10.2.2)

GordonSumerling

Forest,

what hat compression is being applied to the output image? Is it the same as the source. Under the Environments there is a raster storage setting called compression which specifies the compression used on the image. See the help topic

Depending in in the source compression and the resultant image you could see these size changes.

cheers

Gordon

ChrisSmith7

Yes, you're right - the reprojection to web mercator results in smaller cells from the conversion to meter, from foot. Sorry - should have clarified that as an observation rather than the cause.

GabrielUpchurch1

It appears you are confusing comments by two different people. You are exactly correct about the cell sizes since there was a conversion from feet to meters between the two systems. The cells are still the same size so this is not why the datasets are different sizes.

Did you symbolize the NoData as I (Gabriel Upchurch) suggested? My guess is that the dataset was rotated during the transformation which resulted in a large areas of NoData cells being added in the corners.

forestknutsen1

Gabriel Upchurch:
Interesting. It appears there has been a considerable addition of rows and columns. The proportional difference in rows and columns is almost identical to the proportional difference in file sizes. I would symbolize your NoData cells to see if that is where all the extra rows and columns are hiding.

For sure the extra size has to be coming from the increase in rows and columns. But why is this happing? I would think that it would have more or less the same number of rows and columns as the original because:

Which is the cell size that the tool selected.

forestknutsen1

Thanks for the info; however, both images are 8 bit.

ChrisSmith7

Also, looking at your image, raster_cut_small contains more columns and rows; the cells are smaller, and there's a higher level for pyramids. All of this results in more data contained within the file.

GabrielUpchurch1

Interesting. It appears there has been a considerable addition of rows and columns. The proportional difference in rows and columns is almost identical to the proportional difference in file sizes. I would symbolize your NoData cells to see if that is where all the extra rows and columns are hiding.

ChrisSmith7

Esri has a technical article discussing some of these reasons:

28470 - Why does the data bit depth increase when a raster is projected, rotated, or clipped?

This can result in file size changes:

30560 - Why does the file size of my raster image change when I export or create a new image?

Doe mee aan het gesprek

Aangemelde leden kunnen berichten plaatsen, updates volgen en meer. Nieuw hier? Registreer een gratis account.

Snelle Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.