Pixel export performance in Web App: CRF, MRF, ArcGIS and Image Server 10.7.1

Question

I'm creating a Web Application that allows users to visualise multidimensional and spatiotemporal rasters, draw an area on a map and export pixel values from the data-cube into a tabular format.

Ultimately, the application enables users to download pixels values for an area of interest  into a table format, where the location coordinates as well as all parameter attributes for the pixel values are exported for further analysis of Ocean data in a tool of their choice.

I have so far explored a number of options and it seems that some of them are pretty slow when it comes to exporting the pixels.

Visualisation speed is OK.

I am looking for advice to make the pixel export workflow as fast as possible so it can run interactively in the browser.

Thank you for any advice.

Tilmann

I'm using the WABIS2.0 widgets for part of it, in a Web App Builder Dev application,

Image Services

GP/web tools,

all published in ArcGIS Enterprise 10.7.1.

My data cube contains about 1600 rasters right now, but this number will grow over time, as new rasters are added, in LERC compressed MRF format.

WORKFLOW:

Users open Web Application in browser, zooms and pans in map. For their area of interest, they open the GP widget which allows them to draw a polygonal area on the screen. This area is where we want to export all underlying pixels from the data cube into records in a table.

For an area that spans around 5 pixels, the application takes about 2 minutes to export the values. This is clearly too long for any users attention span. What's worse is that an export for larger areas times out (after 30 mins which I've configured).

In ArcGIS Pro, I can use the GP workflow model that I've created to run a "Sample" of a large area (about 1000 pixels) in a fraction of a second, when I use the CRF version of my data cube (the transposed multidimensional raster).

In my web tool, I have had no success whatsoever using the CRF version. (I am getting errors about the dataset not being multidimensional)

When I use the non-transposed (parent-child) Mosaic dataset (set described below), the "Sample" process takes very long.

QUESTIONS:

* Is my setup correct? - see details under Setup Overview in the section below.
* Is it supported in ArcGIS Enterprise 10.7.1 to run this kind of export ("Sample" tool) from a CRF based Mosaic Dataset or Image Service? 
-- "Sample" is the only tool that I'm aware which works on Multidim Rasters and also exports the Coordinates.
* Is there another tool, library or workflow that would be faster for this export?
* I've read about the built-in ability of Image Services to return numpy arrays. How would I leverage that in practice? I presume in a custom Python Raster Function - though how would that look?
* Python Modules such as XArray or Salem could provide another solution - but as they are not aware of CRF format, they can't access the optimized version and I've therefore shied away from implementing them in some form of tool.
* I've read in the ArcGIS documentation about the "built-in P2V" raster function that could be used for Pixel-to-vector conversion (this is from the "Generate Table from Raster Function"  ArcGIS Pro help). Would that be an option? How would I make the request from a Data Cube?
* Are the new capabilities for CRF in ArcGIS Enterprise 10.8 going to fix the trouble with querying CRF? (I haven't been able to get it going in my 10.8 Dev server, but we will update the other ArcGIS Enterprise environments once 10.8.1 is out so we get long-term support for that)

SETUP OVERVIEW:

Data preparation:

* To prepare the data, I've converted them into MRF rasters, using LERC compression (using the "OptimizeRasters" tool)
* Individual Parameter Rasters are added to individual (child) Mosaic Datasets using MDCS, which also calculates Attribute Values that allow for temporal and attribute filtering
* After configuring individual 'children', these are added into a 'parent' Mosaic Dataset as 'table' raster data type.
* Statistics are calculated for the Rasters and Mosaic Datasets
* I update the Multidimensional Information for the 'Parent' Mosaic Dataset, based on the "StdTime' attribute and the "Parameter" or "Variable" attribute
* I create a Referenced Mosaic Dataset from the Parent dataset
* I attach Rendering Rules using the "Manage Processing Templates". The rendering rules apply a filter that queries the parent MDS for an individual parameter/variable and renders the returned rasters using a simple Raster Function that does a "remap" --> "colormap"
* At the end I run "Transpose Multidimensional Raster" to (also) create a CRF-based second version of my multidimensional raster data cube for fast sampling along the time axis.

Services Setup:

* a first image service is published from the Referenced Parent Mosaic Dataset (Image Server 10.7.1.)
* a second image service is publised from the Transposed Mosaic Dataset (Image Server 10.7.1.)
* a GP (Web Tool) is published through ArcGIS Server (10.7.1) that uses "Sample" to export pixels from the CRF-based transposed raster cube (IT DOES NOT QUERY THE IMAGE SERVICE) into an table in the %scratchGDB% or into an in_memory workspace that can then be converted into a CSV file for handing it over into the next workflow (which isn't using ArcGIS) or returned to the user via a link

Application Setup:

Web Appbuilder Dev Edition (2.11) with WABIS2 widgets is used to provide the application framework and I publish the application using Netlify

Emily Windahl‌ Steve Kopp Paul Dodd‌ Peter Becker‌ Vinay Viswambharan‌

MarkMenzel2 · Answer

Hi @TilmannSteinmetz2 ,

Did you get further with the download and sampling of the CRF through time?

We are running 10.8.1 with CRF based multidimensional data (daily timestep for 5 years, 9 variables and 4 ensemble statistics) and have found that the getSample has been not much faster than the NetCDF in a mosaic dataset.

Is your download tool going to allow netCDF as an output?

Cheers,

Mark

TilmannSteinmetz2 · Answer

Thanks very much for your reply, Peter –

That all makes sense to me and re-assuring to know that the setup sounds about correct. Can I just ask to confirm that a GP based web tool (in Python on ArcGIS Server 10.7.1 ) which uses “Sample” can’t work with a CRF-transposed raster file (which is what I’m trying)?

My application demo is here https://scenz-md-is.netlify.app/ - it has three GP tools in the top menu, of which the one that does the export into a file is currently broken, but the rightmost one of the three will run the export (for a short line) and then use the values to run an R-ArcGIS Bridge script in a GP tool do create a chart (so basically only does an additional step at the end. The R script runs super-quick, so the majority of time spent in the tool is the “Sample” process.

All rasters, mosaic datasets and the web map are in NZTM (New Zealand Transverse Mercator) projection and rasters are all pixel-aligned.

Cheers,

Tilmann

PeterBecker · Answer

There are many different method to achieve this and the optimum choice is dependent on factors such as whether the source is pixel aligned, the total size of the data and type of storage (Local or cloud).

The method you are using works well and is similar to what is used in the Hycom sample.

.

In this case the source is similar to what you mention, a MRF referenced by a mosaic dataset.

What you are doing is not incorrect, but we know there are limitations to this as the datasets get larger.

This use of a Mosaic dataset and individual rasters per slice is not optimized for the fact that in this case all the pixels are aligned in a single coordinate system. (Remember mosaic datasets can many different coordinate systems simultaneously). Also the data is stored as pixel blocks. To get a temporal profile requires reading many pixel blocks from storage. If the storage is fast and you don't have too many layers then this is not an issue.

In your case you are referring to over 1000 layers, so such as solution would need to read over a thousand pixel blocks for any profile location. (Not optimum).

The solution as you have seen is to use a CRF with Transpose option. The CRF is optimized for the storage of such pixel aligned datasets and the transpose re-arranges the storage of the pixels such that a query for a profile need only make minimal disk access. You would still manage your data by creating a Mosaic dataset.

With ArcGIS Image Server 10.8 you can directly publish a Transposed CRF and the Sample function will then near instantaneously return you the pixel for a point or small area of interest. You can you this for a profile or extract to a table of choice.

One drawback with Pro 2.5/10.8 with CRFs is that you can not append to a CRF so that when you get a new dataset  you need to recreate the CRF. (But such an append is available in upcoming 2.5/10.8.1).

What I need to check on is how WABIS provides access to this profile capability. I'll check and get back.