<\/HEAD>
Do you also struggle with the ever-increasing amount of raster data? We do! For example, with the AHN data: the current elevation map of the Netherlands. How do we store that as cost-effectively as possible, but at the same time offer it quickly and scalably as a web service? These are challenging puzzles to work on, but also educational and they provide new insights. I am happy to share the insights gained through this article.<\/SPAN><\/P> <\/SPAN><\/P>The Current Elevation Database of the Netherlands (AHN) has multiple editions. At this moment, the Dutch government is adopting the 3rd<\/SUP> version: AHN3 (over multiple years). Several areas of the Netherlands are already available and in the coming years, the remaining areas will follow. So every year more data is added and before you know it, you go from many GBs of data to TBs.<\/SPAN><\/P> <\/SPAN><\/P>We were looking for a smart way to store these rasters from the AHN. In our search, we came across the Meta Raster Format (MRF). This is a raster format invented by NASA, which Esri uses in the context of serving rasters on the web and via cloud infrastructure.<\/SPAN><\/P> <\/SPAN><\/P>In MRF, several components of the traditional raster are split into separate files and the data is logically organized. Through an index file, the sizes and geometric organization of the raster pixels are stored and a metadata file provides the most important information, such as the number of rows and columns, the data type, the tile setup (tiling), and projection. This means that not the entire raster file needs to be read or sent. Because of this technique, it is even possible to separate these different types of files and place them on different storage volumes. Very handy for scaling. The index files and metadata files can then be placed on a fast SSD drive for example, and the data files on a slower drive. The index files allow quick determination of which data files need to be queried and the metadata file indicates how to interpret or place these in space. This ensures that performance is maintained, but not all files need to be on fast storage volumes, allowing you to save costs.<\/SPAN><\/P> <\/SPAN><\/P>
<\/SPAN><\/P><\/P>In addition to the MRF file format, we also used a relatively new compression method. With compression, files often become smaller in storage size, but for serving data over the web, this means that data must be unpacked each time to be used. Current techniques such as JPEG2000, PNG, Deflate, and LZW can be used well in combination with MRF but have disadvantages regarding volume reduction or CPU usage. Esri has developed a new compression method called Limited Error Raster Compression (LERC). LERC was specifically developed to facilitate fast compression/decompression and low CPU usage. LERC can be used for both 'Lossless' (which does not alter raster pixels) and 'Lossy' (which alters raster pixels to optimize compression). If tolerance is set to 0, lossless compression is applied; if higher than zero, lossy compression is applied. We applied this last one to AHN with a tolerance of 0.01 m. This keeps each cell's value well within stochastic deviation (5 cm for AHN2/3), but still achieves compression of about a factor of 5-7 compared to GeoTIFF without compression and a factor of 3-4 compared to GeoTIFF with LZW compression. Of course, it is possible to make the data even smaller, but then pixel values will vary and this is not desired for AHN.<\/SPAN><\/P> <\/SPAN><\/P>Long story short... the following table makes the point about storage clear:<\/SPAN><\/P> <\/SPAN><\/P> <\/P><\/TD>Storage as GeoTIFF (without compression)<\/P><\/TD>Storage as GeoTIFF <\/SPAN><\/P>with LZW compression<\/SPAN><\/P><\/TD>Storage as MRF with LERC compression<\/P><\/TD><\/TR>< P style = " margin : ">AHN2 i< / P >< / TD >< TD style = "border - top : none ; border - left : none ; border - bottom : solid windowtext ; border - right : solid windowtext ; padding : 5 .4 pt ; height : 25 px ; " width = "175" >< P style = " margin : ">408 GB< / P >< / TD >< TD style = "border - top : none ; border - left : none ; border - bottom : solid windowtext ; border - right : solid windowtext ; padding : 5 .4 pt ; height : 25 px ; " width = "175" >< P style = " margin : ">272 GB< / P >< / TD >< TD style = "border - top : none ; border - left : none ; border - bottom : solid windowtext ; border - right : solid windowtext ; padding : 5 .4 pt ; height : 25 px ; " width = "210" >< P style = " margin : ">72 .7 GB< / P >< / TD >< / TR >< TR style = "height :25px" >< TD style = "border - right :solid windowtext;border - bottom :solid windowtext;border - left :solid windowtext;border - image :initial;border - top:none;padding :5 .4 pt;height25px" width=245 >< P >AHN2 r< / P >< / TD >< TD style = "border - top:none;border - left:none;border - bottom :solid windowtext;border - right :solid windowtext;padding :5 .4 pt;height25px" width=175 >< P >465 GB (estimated)< / P >< / TD >< TD style = "border - top:none;border - left:none;border - bottom :solid windowtext;border - right :solid windowtext;padding :5 .4 pt;height25px" width=175 >< P >316 GB< / P >< / TD >< TD style = "border - top:none;border - left:none;border - bottom :solid windowtext;border - right :solid windowtext;padding :5 .4 pt;height25px" width=210 >< P >97 .1 GB< / P >< / TD >< / TR >< TR style = "height50px" >< TD style = "border - right :solid windowtext;border - bottom :solid windowtext;border - left :solid windowtext;border - image :initial;border - top:none;padding :5 .4 pt;height50px" width=245 >< P >AHN3 i (approximately70% complete for NL)< / P >< / TD >< TD style = "border - top:none;border - left:none;border - bottom :solid windowtext;border - right :solid windowtext;padding :5 .4 pt;height50px" width=175 >< P >391 GB (estimated)< / P >< / TD >< TD style = "border - top:none;border - left:none;border - bottom :solid windowtext;border - right :solid windowtext;padding :5 .4 pt;height50px" width=175 >< P >241 GB< / P >< / TD >< TD style = "border - top:none;border - left:none;border - bottom :solid windowtext;border - right :solid windowtext;padding :5 .4 pt;height50px" width=210 >< P >46 .3 GB< / P >< / TD >< / TR >< TR style = "height50px" >< TD style = "border - right :solid windowtext;border - bottom :solid windowtext;border - left :solid windowtext;border - image :initial;border - top:none;padding :5 .4 pt;height50px" width=245 >< P >AHN3 r (approximately70% complete for NL)< / P >< / TD >< TD style = "border-top:none;border-left:none;border-bottom: 1pt solid windowtext; padding: 0in 5.4pt; height: 50px;" width="175">276 GB
61.1 GB
For more information about MRF and LERC, also see the article by Peter Becker, Product Manager for Imagery at Esri Inc.
