SDO_GEORASTER support within Arcsde 10.1 �?? why only blocksize 128x128 is supported!

Question

When loading data into Oracle SDO_GEORASTER using GDAL, we can choose different blocksize. When working with database storage for raster, Oracle team mentions that it is better to user bigger block size from 256x256, 512x512 or even 1024x1024. I am able to load my dataset into Oracle, and view it via Oracle Mapviewer, or QGIS but when trying with Esri, I can only use 128x128 block size.
If I connect to 256x256 table, I see my raster in Arc Catalog, but black lines appears depending used scale. When using 512x512, this is not possible to view it.
I played with RASTERBUFSIZE increasing memory, but nothing change. I am using Esri 10.1 �?? patchset 1 (last available).
The goal of increasing blocksize is to minimize number of blocks, and decrease database access. Rule about 128x128 blocksize was good for files, but is not anymore valid for database storage. Also, loading time when using that blocksize is really increasing. I have 1TB of data to load!
Any feedback on that?
Why Esri does not support bigger blocksize?

Olivier

MarcoBoeringa · Answer

I also used the following command to define RASTERBUFSIZE
sdeconfig -o alter -v RASTERBUFSIZE=1048576 -u sde -p SDE@INFRASIG11G_TEST -i sde:oracle11g

To be honnest, this is not clear how to pick up the best value for this parameter. Can you provide a sample how to calculate using this source file. But I tried also with higher values but nothing works.

If I understand it well from the Help text below:

"The uncompressed size of a raster tile can be determined by multiplying the tile width by the tile height and adjusting for the pixel depth. For instance, if you use the default values of 128 by 128 for the tile width and tile height, the product of these two values is 16,384. Multiply this product by the pixel depth factor from the table below to obtain the raster tile's uncompressed size. If the pixel depth is 32, the uncompressed size would be calculated as 65,536 bytes. This is well under the 200 KB default value. However, if you decide to increase the tile size to 256 by 256, it would increase the uncompressed size to 262,144 bytes."

each band of your 4 band imagery would occupy 262Kb in the buffer. So 4x262 would already mean a minimum required buffer size of 1048Kb, the one you set. But I imagine having some "overhead" is necessary too, so I would try setting a bigger value than that for a tile size of 256x256.

Anyway, I have heard of issues with SDO_GEORASTER before, I don't know if ESRI actually ironed out all the issues. I don't have a setup available to test, maybe someone else with real experience with native SDO_GEORASTER on Oracle will join the thread and be able to help you out.

OlivierDubois · Answer

What type of imagery are you using (how many bands, pixel depth per band - 8bit, 16bit etc.) and what values for RASTERBUFSIZE did you try? If you are using imagery that has many bands or high pixel depth, you could still run out of ArcSDE buffer memory, even when adjusting buffer size. Have you build statistics and pyramids?

Please find below a sample file used using gdalinfo.
This raster contains 4 bands, 8 bits (byte) per band.
     [echo] Command used : gdalinfo ./tempfiles/00740081.tif 
     [exec] Driver: GTiff/GeoTIFF
     [exec] Files: ./tempfiles/00740081.tif
     [exec] Size is 8000, 8000
     [exec] Coordinate System is:
     [exec] PROJCS["Belge 1972 / Belgian Lambert 72",
     [exec]     GEOGCS["Belge 1972",
     [exec]         DATUM["Reseau_National_Belge_1972",
     [exec]             SPHEROID["International 1924",6378388,297.0000000000014,
     [exec]                 AUTHORITY["EPSG","7022"]],
     [exec]             AUTHORITY["EPSG","6313"]],
     [exec]         PRIMEM["Greenwich",0],
     [exec]         UNIT["degree",0.0174532925199433],
     [exec]         AUTHORITY["EPSG","4313"]],
     [exec]     PROJECTION["Lambert_Conformal_Conic_2SP"],
     [exec]     PARAMETER["standard_parallel_1",51.16666723333333],
     [exec]     PARAMETER["standard_parallel_2",49.8333339],
     [exec]     PARAMETER["latitude_of_origin",90],
     [exec]     PARAMETER["central_meridian",4.367486666666666],
     [exec]     PARAMETER["false_easting",150000.013],
     [exec]     PARAMETER["false_northing",5400088.438],
     [exec]     UNIT["metre",1,
     [exec]         AUTHORITY["EPSG","9001"]],
     [exec]     AUTHORITY["EPSG","31370"]]
     [exec] Origin = (162000.000000000000000,150000.000000000000000)
     [exec] Pixel Size = (0.250000000000000,-0.250000000000000)
     [exec] Metadata:
     [exec]   AREA_OR_POINT=Area
     [exec] Image Structure Metadata:
     [exec]   INTERLEAVE=PIXEL
     [exec] Corner Coordinates:
     [exec] Upper Left  (  162000.000,  150000.000) 
     [exec] Lower Left  (  162000.000,  148000.000) 
     [exec] Upper Right (  164000.000,  150000.000) 
     [exec] Lower Right (  164000.000,  148000.000) 
     [exec] Center      (  163000.000,  149000.000) 
     [exec] Band 1 Block=8000x1 Type=Byte, ColorInterp=Red
     [exec]   Mask Flags: PER_DATASET ALPHA 
     [exec] Band 2 Block=8000x1 Type=Byte, ColorInterp=Green
     [exec]   Mask Flags: PER_DATASET ALPHA 
     [exec] Band 3 Block=8000x1 Type=Byte, ColorInterp=Blue
     [exec]   Mask Flags: PER_DATASET ALPHA 
     [exec] Band 4 Block=8000x1 Type=Byte, ColorInterp=Alpha

I also used the following command to define RASTERBUFSIZE
sdeconfig -o alter -v RASTERBUFSIZE=1048576 -u sde -p SDE@INFRASIG11G_TEST -i sde:oracle11g

To be honnest, this is not clear how to pick up the best value for this parameter. Can you provide a sample how to calculate using this source file. But I tried also with higher values but nothing works.

Also, sdo_georaster data are well stored and defined. Pyramids and statistics are correctly done. I can see data with others tools than Esri without problems (and using pyramids).
Do you get the "SE_RASTER_BUFFER_TOO_SMALL (-294)" error as mentioned in the Help topic below

Loading large raster datasets into ArcSDE

No, this is working. I had issues when RASTERBUFSIZE was too low but now It is working with 128x128. I can see partial data using 256x256, but blacklines appears and images in block seems to be stretched.
Do you have any specific reason to use GDAL for the loading of your raster data and not using ArcGIS? Have you tried loading using ArcGIS?

We never achieved to load data using Arcgis in sdo_georaster format. I have an open bug with Esri Belux, they can reproduce the issue. When loading with GDAL in 128x128x1 everything is working, we can see the data in Esri Arccatalog, Oracle Mapbuilder and QGIS. But I can't use other tilesize with Esri.

MarcoBoeringa · Answer

If I connect to 256x256 table, I see my raster in Arc Catalog, but black lines appears depending used scale. When using 512x512, this is not possible to view it.
I played with RASTERBUFSIZE increasing memory, but nothing change. I am using Esri 10.1 �?? patchset 1 (last available).

What type of imagery are you using (how many bands, pixel depth per band - 8bit, 16bit etc.) and what values for RASTERBUFSIZE did you try? If you are using imagery that has many bands or high pixel depth, you could still run out of ArcSDE buffer memory, even when adjusting buffer size. Have you build statistics and pyramids?

Do you get the "SE_RASTER_BUFFER_TOO_SMALL (-294)" error as mentioned in the Help topic below

Loading large raster datasets into ArcSDE

Do you have any specific reason to use GDAL for the loading of your raster data and not using ArcGIS? Have you tried loading using ArcGIS?