topic Re: Converting 32-bit TIFF to 16-bit erroneous values in Imagery and Remote Sensing Questions

Converting 32-bit TIFF to 16-bit erroneous values

JohnWall — Tue, 06 Feb 2018 18:09:55 GMT

I have several thousand TIFF files that I need to convert from 32-bit to 16-bit, but the Copy Raster geoprocessing tool appears to be providing erroneous results.

As an example, I have a TIF in 32-bit format with values ranging 0.0241 to 0.0674. When I use Copy Rater and set the "NoData Value" to blank, "Pixel Type" to 16-bit unsigned, set the "Scale Pixel Value" to true, I get a TIFF with values ranging 0 to 32768, but the image appears blank no matter what histogram stretch I use. I have tried Calculating Statistics within ArcCatalog with no results. When I set "Scale Pixel Value" to false, I get values ranging 0 to 1 with an image that appears blank.

I am using ArcGIS 10.3.1.

Is there something I am missing?

Re: Converting 32-bit TIFF to 16-bit erroneous values

JieZhang — Tue, 06 Feb 2018 19:19:44 GMT

Hi John,

Is it possible for you to upload a small sample image here? We can take a look. You didn't miss anything based on your description.

Thanks

Jie

Re: Converting 32-bit TIFF to 16-bit erroneous values

JohnWall — Tue, 06 Feb 2018 22:30:19 GMT

Hi Jie,

Here's the GeoTIFF.

Thanks!

Re: Converting 32-bit TIFF to 16-bit erroneous values

DanPatterson_Retired — Tue, 06 Feb 2018 22:47:10 GMT

could be software version

In PRO, if I got it right

Re: Converting 32-bit TIFF to 16-bit erroneous values

CodyBenkelman — Tue, 06 Feb 2018 23:06:24 GMT

John

It always helps if you can give us a complete story. e.g. You did what many do NOT do (told us the version are you using), but what are your overall objectives? What is the data? Why are you converting thousands of files?

At risk of changing the subject, I am concerned any time I see questions about "processing large # of images" and "creating a lot of outputs". That's especially true in a case like this, where you're starting with floating point data (where I presume the pixel value is some absolute measurement - I would have guessed orthometric height although your range above "0.0241 to 0.0674" sounds odd) and converting for some reason, to generate files with 16 bit values that now do NOT represent the true data values but must have some legend or lookup table if users ever need to determine the (approximate) original values...

Are the end users for these 16 bit files using some other software? Have you considered putting all into one single mosaic dataset and applying a raster function to rescale from F32 to U16? Your project might be completely done in an hour...

Cody B.

Re: Converting 32-bit TIFF to 16-bit erroneous values

JohnWall — Tue, 06 Feb 2018 23:29:50 GMT

Hello Cody,

The image was created in Pix4D from UAV images recorded with a MicaSense RedEdge sensor. Although the camera records in 16-bit, Pix4D created 32-bit GeoTIFF outputs. I should be able to switch the output to 16-bit in Pix4D, but I have not been able to find that setting in the version of the software I have and I think that issue is maybe best left to a different forum. If that is something I should include, please let me know. Due to the bit-depth issue, I thought I should be able to convert the values in ArcGIS and be good to go.

I am converting thousands of files from roughly 70 surveys that were carried out by someone before me on a university/research project. These surveys cover a series of turfgrass plots and surveys were repeated on multiple days with the aim of observing a blight taking over the grass fields. Some surveys have RGB composite rasters/orthophotos/images while others have individual reflectance rasters which I was planning to convert to 16-bit and create a composite raster (R,G,B, NIR and RedEdge).

I thought the 0.0241 to 0.0674 sounded odd! I will go over the workflow I was provided with and see if I get different values from reprocessing.

The main reason I have been tasked with converting from 32- to 16-bit is the next person is suppose to carry out supervised and unsupervised image classification in ERDAS imagine. They picked that software since it is the package they are familiar with for that kind of work. They are hoping that the spectral signal in the images I process will be enough to classify the turfgrass blight automatically.

Please let me know if I did not answer your questions fully.
Thank you,

John

Re: Converting 32-bit TIFF to 16-bit erroneous values

JohnWall — Tue, 06 Feb 2018 23:31:18 GMT

Hi Dan,

Thank you for your note! For other reasons I would like to stay in 10.3.1, personally. But, I'll give ArcGIS Pro a shot as well.

Thank you,

John

Re: Converting 32-bit TIFF to 16-bit erroneous values

CodyBenkelman — Wed, 07 Feb 2018 17:27:04 GMT

John

this makes sense.

I'd encourage you to try doing all of this in ArcGIS Pro. (If you don't have Pro now, you can get a trial copy)

It may be possible that a change in your workflow could output 16 bit integers from Pix4D, but if you have 70 surveys you don't want to have to reprocess all of those through Pix4D.

You should not need to convert to 16bit. If you creating a mosaic dataset of all the imagery, you can treat it as one logical collection. The mosaic dataset can also combine any desired bands into a multispectral view (e.g. 4,1,2 for color IR, 4 & 1 for NDVI, 5 & 1 for red-edge-NDVI, etc.), and this is done instantly via a function, not a process you need to run and create more output files.

For the classification, you or colleagues can use advanced classification tools and wizard in the Image Analyst Extension or Spatial Analyst. (also available as a free trial for ArcGIS Pro). This can use any bit depth, or you can apply an arithmetic function prior to classification to convert (on the fly) to 16 bit.

If you really need to proceed with converting to 16bit, then you can use OptimizeRasters.

Download from http://esriurl.com/OptimizeRasters and use following command

c:\Image_Mgmt_Workflows\OptimizeRasters\OptimizeRasters.py -config=Imagery_to_TIF_float2int.xml -input=c:\tmp\Q\inputdirectory -output=c:\tmp\Q\outputdirectory

(See http://s3.amazonaws.com/ImageManagementWorkflowsTeam/OptimizeRasters/Imagery_to_TIF_float2int.xml )

Re: Converting 32-bit TIFF to 16-bit erroneous values

JohnWall — Wed, 07 Feb 2018 18:22:16 GMT

Hi Cody,

Thank you for your suggests. I'll run these past the rest of the group.

At the risk of sounding like I've skipped over something, I understand how a mosaic can be created in ArcGIS, but I do not know how ortho images would be created from UAV data (i.e. non-nadir). Is there a workflow for that? Is that something that has been added between 10.3.1 and Pro?

Thank you,

John

Re: Converting 32-bit TIFF to 16-bit erroneous values

JieZhang — Wed, 07 Feb 2018 20:09:03 GMT

Hi John,

This is a bug of this tool in this version of ArcMap.

If you have to stay at 10.3.1 and you have to use this tool, I suggest you submit a support case through this link Esri Support Contact Support. There is workaround to use the stretch raster function through the image analysis window and change the output pixel type. However you cannot automate this through scripting.

It would be the best if you can upgrade either to Pro or newer version of ArcMap.

Jie

Re: Converting 32-bit TIFF to 16-bit erroneous values

CodyBenkelman — Fri, 09 Feb 2018 06:25:28 GMT

John

There's a GP tool from long ago that will create mosaic files on disk from multiple inputs, but that's not a Mosaic Dataset which is our modern database structure for accessing and processing imagery on the fly, without duplicating data in derived image products. If you aren't familiar, see What is a mosaic dataset?—Help | ArcGIS for Desktop

Note the Mosaic Dataset is for data that already includes georeferencing information, or adequate camera/satellite model and image metadata for on-the-fly orthorectification.

For raw single frame drone images, a photogrammetric workflow for processing (to create orthomosaics, DSMs, point clouds & more) is available in http://esri.com/drone2map which is a standalone app (does not require ArcGIS). It is sold at a price that is very competitive to similar tools in the marketplace (with numerous points of direct interface with ArcGIS & ArcGIS Online). Or, if you have a current license for ArcGIS Desktop, ArcGIS Pro 2.1 includes a drone camera workflow in the Ortho Mapping tools (built into Pro, no add-on cost).

There is a workflow for centralized data management of massive numbers of Drone2Map projects in the imagery workflows. See the free script at http://esriurl.com/D2Mmanagement

For drones capturing video, we have the Full Motion Video add-in for working with geotagged video, in both map space or with GIS features projected into the video. http://esri.com/fmv. (Note this isn't just for drones - it was originally created for professional/military grade video systems but can support drone video through a postflight geoprocessing tool called the "Video Multiplexer")

ArcGIS has quite a bit to offer for drone imagery, and more is coming.

Cody