topic Re: Trouble interpolating 7M points in ArcGIS Spatial Analyst Questions

Trouble interpolating 7M points

SimonJackson — Sat, 10 Jun 2017 00:57:31 GMT

I have a layer with over 7 million records. This is in a fine mesh grid (output from a 3rd party modelling package.

The source data is a simple table in SQL Server containing a geometry column (4326).

I need to interpolate this using either Natural Neighbour or IDW using a variable in the dataset.

I have used a smaller area and have been happy with the results I have been getting.

I have replicated the below in ArcGIS Pro 1.4 and 2.0 beta and also in ArcMap 10.5.

I have 16GB RAM and a decent CPU and a dedicated GPU.

Using Spatial Analyst IDW

Using the Query Layer

Completes ~5 minutes

Result is an empty raster with no data.

Using Enterprise GeoDatabase

Completes ~2 minutes

Result is an empty raster with no data.

File GeoDatabase

Completes ~2 minutes

Result is an empty raster with no data.

Using Spatial Analyst Natural Neighbour

Query Layer ~5 minutes = Empty Raster

File Geodatabase 38 seconds = Empty Raster

I also have Geostatistical Analyst, and did see it had some options for parallel processing, but I also spotted that IDW does not have a search distance which I need.

Can anyone give me some pointers on how to interpolate this dataset?

Re: Trouble interpolating 7M points

DanPatterson_Retired — Sat, 10 Jun 2017 02:32:17 GMT

When you say 'fine mesh' that suggests to me that the points are spaced on a grid... let's say that the spacing in the X and Y direction is 1 meter... Just convert it directly to a raster using that as a cell size... why interpolate? Maybe you want something in the 0.5 meter cell size... I would then have a look at Resample first. It will sure save a lot of grief and you will retain your original point values depending on the method chosen.

Re: Trouble interpolating 7M points

JayantaPoddar — Sat, 10 Jun 2017 03:09:46 GMT

Did you try using a standalone python script? Natural Neighbor—Help | ArcGIS Desktop

Re: Trouble interpolating 7M points

DanPatterson_Retired — Sat, 10 Jun 2017 05:19:41 GMT

Just noticed your coordinate system is 4326... isn't that geographic? Did you project it first since you wanted a 10 m cell size or were you hoping the projection of the data frame/map would be used assuming it is in projected coordinates

Re: Trouble interpolating 7M points

SimonJackson — Sat, 10 Jun 2017 05:33:31 GMT

Hi Dan Patterson‌.

Coordinate system

With the query layer (4326) I set an environmental variable to output the results in a projected coordinate system.

With the Enterprise GDB/File GDB, the data had already been projected into a pcs.

Point to Raster

Good suggestion. The 'cell size' of the point layer is 40m.

Would have been desirable to interpolate to 5-10m, but resampling might be an approach there. I assumed (disregarding size of the dataset for now) that Nearest Neighbour would have been the optimal approach for this?

And what if the orientation of the points is not exactly linear, would this not cause an issue?

Hmm. Just tried running the tool and it produced an empty output. Will investigate.

EDIT

I got it to output a layer, but it does not convert all the points.

Interestingly, if I apply a graduated renderer, it ends up not showing all the points (the same as what does not get created by the point to raster).

The points that have not been rendered have values assigned to them and are valid geometries.

Does this provide a clue into what could be causing the issue?

Below: Layer with a graduated renderer. Layer with a simple renderer (squares). A simple result of the point to raster tool.

The Point to Raster output matches the same values not rendered by the unique renderer.

All those datapoints have a value behind them.

Perhaps when it tries to come up with a classification scheme, a graduated renderer stops scanning a table after a certain number of records?

From an interpolation/point to raster perspective, seems like it might be hitting this same issue and not outputting the correct result?

Point to Raster output full extent:

This does not look correct in several areas (that match up to the areas that fail to render with the graduated renderer)

Re: Trouble interpolating 7M points

GünterDörffel — Sat, 10 Jun 2017 12:35:45 GMT

Hi Simon,

Dont know about the attribution of your points, but especially if there is/are more than one attribute and you want to be flexible on selection, 'layering' and using it as raster or feature to solve your interpolation issue, probably try to convert the data to a multidimensional NetCDF dataset (see help) ... and then use it as a raster layer created from the NetCDF data - this will include your options for setting spacing/resolution and doesnt require the points to be evenly spread to start with. Would be interesting to hear if this works for you. NetCDF is powerful and nicely integrated with ArcGIS nowadays.

Guenter

Re: Trouble interpolating 7M points

XanderBakker — Sat, 10 Jun 2017 13:50:33 GMT

Have you tried using a Terrain Dataset? See What is a terrain dataset?—Help | ArcGIS Desktop . Try that first.

If that does not work you could also go for a TIN. See What is a TIN surface?—Help | ArcGIS Desktop and convert that to raster.

If that does not work either, you could convert the points to text and use LAStools (see LAStools: converting, filtering, viewing, processing, and compressing LIDAR data in LAS format , check licensing, but some tools used to be free ) to convert the TXT to LAS and create a LAS dataset (Creating a LAS dataset—Help | ArcGIS Desktop ) and convert that to raster LAS Dataset To Raster—Help | ArcGIS Desktop

For large datasets ArcGIS enterprise come with Raster Analytics Perform raster analysis—Documentation | ArcGIS Enterprise , but let's be serious 7M points is too small for that...

Re: Trouble interpolating 7M points

XanderBakker — Sat, 10 Jun 2017 13:56:57 GMT

And here is an "old" document on interpolation methods, however, the info is still valid:

https://www.esri.com/news/arcuser/0704/files/interpolating.pdf

And please do convert you data into projected coordinates before interpolation and check the environment settings, since the cellsize (1 meters) might be conflicting with the geographic coordinates and yield cell sizes of 5 or 10 degrees. What is a geoprocessing environment setting?—Help | ArcGIS Desktop

Re: Trouble interpolating 7M points

curtvprice — Sun, 11 Jun 2017 00:36:29 GMT

I totally second the motion on terrain -- this data structure is designed for fully specified point data like photogrammetric or lidar data.

Xander, FYI TINs are an old ArcInfo data structure and very limited as to # of points, TIN limitations led to the development of geodatabase terrains.

Re: Trouble interpolating 7M points

curtvprice — Sun, 11 Jun 2017 00:38:43 GMT

but I also spotted that IDW does not have a search distance which I need.

IDW search distance is specified using the parameter search_radius.

Re: Trouble interpolating 7M points

SimonJackson — Sun, 11 Jun 2017 10:18:19 GMT

Interesting. After creating the surface, I will be doing some raster calculations against this layer and multiple other raster datasets. Any issues using a terrain dataset for this?

Re: Trouble interpolating 7M points

SimonJackson — Sun, 11 Jun 2017 10:19:50 GMT

I have been contemplating if a NetCDF would be useful in other areas of this study (the data is also temporal).
In this instance, there is just one variable within this point dataset that I need to create a surface from.
However, will have a bigger think about this.

Re: Trouble interpolating 7M points

XanderBakker — Sun, 11 Jun 2017 15:33:21 GMT

You would use Terrain To Raster—Help | ArcGIS Desktop to create the DEM in raster format and carry out the analysis like with any other method.