I'm creating a Web Application that allows users to visualise multidimensional and spatiotemporal rasters, draw an area on a map and export pixel values from the data-cube into a tabular format.
Ultimately, the application enables users to download pixels values for an area of interest into a table format, where the location coordinates as well as all parameter attributes for the pixel values are exported for further analysis of Ocean data in a tool of their choice.
I have so far explored a number of options and it seems that some of them are pretty slow when it comes to exporting the pixels.
Visualisation speed is OK.
I am looking for advice to make the pixel export workflow as fast as possible so it can run interactively in the browser.
Thank you for any advice.
Tilmann
I'm using the WABIS2.0 widgets for part of it, in a Web App Builder Dev application,
Image Services
GP/web tools,
all published in ArcGIS Enterprise 10.7.1.
My data cube contains about 1600 rasters right now, but this number will grow over time, as new rasters are added, in LERC compressed MRF format.
WORKFLOW:
Users open Web Application in browser, zooms and pans in map. For their area of interest, they open the GP widget which allows them to draw a polygonal area on the screen. This area is where we want to export all underlying pixels from the data cube into records in a table.
For an area that spans around 5 pixels, the application takes about 2 minutes to export the values. This is clearly too long for any users attention span. What's worse is that an export for larger areas times out (after 30 mins which I've configured).
In ArcGIS Pro, I can use the GP workflow model that I've created to run a "Sample" of a large area (about 1000 pixels) in a fraction of a second, when I use the CRF version of my data cube (the transposed multidimensional raster).
In my web tool, I have had no success whatsoever using the CRF version. (I am getting errors about the dataset not being multidimensional)
When I use the non-transposed (parent-child) Mosaic dataset (set described below), the "Sample" process takes very long.
QUESTIONS:
- Is my setup correct? - see details under Setup Overview in the section below.
- Is it supported in ArcGIS Enterprise 10.7.1 to run this kind of export ("Sample" tool) from a CRF based Mosaic Dataset or Image Service?
-- "Sample" is the only tool that I'm aware which works on Multidim Rasters and also exports the Coordinates. - Is there another tool, library or workflow that would be faster for this export?
- I've read about the built-in ability of Image Services to return numpy arrays. How would I leverage that in practice? I presume in a custom Python Raster Function - though how would that look?
- Python Modules such as XArray or Salem could provide another solution - but as they are not aware of CRF format, they can't access the optimized version and I've therefore shied away from implementing them in some form of tool.
- I've read in the ArcGIS documentation about the "built-in P2V" raster function that could be used for Pixel-to-vector conversion (this is from the "Generate Table from Raster Function" ArcGIS Pro help). Would that be an option? How would I make the request from a Data Cube?
- Are the new capabilities for CRF in ArcGIS Enterprise 10.8 going to fix the trouble with querying CRF? (I haven't been able to get it going in my 10.8 Dev server, but we will update the other ArcGIS Enterprise environments once 10.8.1 is out so we get long-term support for that)
SETUP OVERVIEW:
Data preparation:
- To prepare the data, I've converted them into MRF rasters, using LERC compression (using the "OptimizeRasters" tool)
- Individual Parameter Rasters are added to individual (child) Mosaic Datasets using MDCS, which also calculates Attribute Values that allow for temporal and attribute filtering
- After configuring individual 'children', these are added into a 'parent' Mosaic Dataset as 'table' raster data type.
- Statistics are calculated for the Rasters and Mosaic Datasets
- I update the Multidimensional Information for the 'Parent' Mosaic Dataset, based on the "StdTime' attribute and the "Parameter" or "Variable" attribute
- I create a Referenced Mosaic Dataset from the Parent dataset
- I attach Rendering Rules using the "Manage Processing Templates". The rendering rules apply a filter that queries the parent MDS for an individual parameter/variable and renders the returned rasters using a simple Raster Function that does a "remap" --> "colormap"
- At the end I run "Transpose Multidimensional Raster" to (also) create a CRF-based second version of my multidimensional raster data cube for fast sampling along the time axis.
Services Setup:
- a first image service is published from the Referenced Parent Mosaic Dataset (Image Server 10.7.1.)
- a second image service is publised from the Transposed Mosaic Dataset (Image Server 10.7.1.)
- a GP (Web Tool) is published through ArcGIS Server (10.7.1) that uses "Sample" to export pixels from the CRF-based transposed raster cube (IT DOES NOT QUERY THE IMAGE SERVICE) into an table in the %scratchGDB% or into an in_memory workspace that can then be converted into a CSV file for handing it over into the next workflow (which isn't using ArcGIS) or returned to the user via a link
Application Setup:
Web Appbuilder Dev Edition (2.11) with WABIS2 widgets is used to provide the application framework and I publish the application using Netlify
Emily Windahl Steve Kopp Paul Dodd Peter Becker Vinay Viswambharan
