Trouble interpolating 7M points

[SimonJackson]

I have a layer with over 7 million records.  This is in a fine mesh grid (output from a 3rd party modelling package.

The source data is a simple table in SQL Server containing a geometry column (4326).

I need to interpolate this using either Natural Neighbour or IDW using a variable in the dataset.

I have used a smaller area and have been happy with the results I have been getting.

I have replicated the below in ArcGIS Pro 1.4 and 2.0 beta and also in ArcMap 10.5.

I have 16GB RAM and a decent CPU and a dedicated GPU.

Using Spatial Analyst IDW

Using the Query Layer

Completes ~5 minutes

Result is an empty raster with no data.

Using Enterprise GeoDatabase

Completes ~2 minutes

Result is an empty raster with no data.

File GeoDatabase

Completes ~2 minutes

Result is an empty raster with no data.

Using Spatial Analyst Natural Neighbour

Query Layer ~5 minutes = Empty Raster

File Geodatabase 38 seconds = Empty Raster

I also have Geostatistical Analyst, and did see it had some options for parallel processing, but I also spotted that IDW does not have a search distance which I need.

Can anyone give me some pointers on how to interpolate this dataset?  

[DanPatterson_Retired]

When you say 'fine mesh' that suggests to me that the points are spaced on a grid... let's say that the spacing in the X and Y direction is 1 meter... Just convert it directly to a raster using that as a cell size... why interpolate?  Maybe you want something in the 0.5 meter cell size... I would then have a look at Resample first.  It will sure save a lot of grief and you will retain your original point values depending on the method chosen.

[JayantaPoddar]

Did you try using a standalone python script? Natural Neighbor—Help | ArcGIS Desktop 

[DanPatterson_Retired]

Just noticed your coordinate system is 4326... isn't that geographic?  Did you project it first since you wanted a 10 m cell size or were you hoping the projection of the data frame/map would be used assuming it is in projected coordinates

[SimonJackson]

Hi Dan Patterson‌.  

Coordinate system

With the query layer (4326) I set an environmental variable to output the results in a projected coordinate system.

With the Enterprise GDB/File GDB, the data had already been projected into a pcs.

Point to Raster

Good suggestion. The 'cell size' of the point layer is 40m.  

Would have been desirable to interpolate to 5-10m, but resampling might be an approach there.  I assumed (disregarding size of the dataset for now) that Nearest Neighbour would have been the optimal approach for this?

And what if the orientation of the points is not exactly linear, would this not cause an issue?

Hmm. Just tried running the tool and it produced an empty output.  Will investigate.

EDIT

I got it to output a layer, but it does not convert all the points.

Interestingly, if I apply a graduated renderer, it ends up not showing all the points (the same as what does not get created by the point to raster).

The points that have not been rendered have values assigned to them and are valid geometries. 

Does this provide a clue into what could be causing the issue?

Below: Layer with a graduated renderer.  Layer with a simple renderer (squares). A simple result of the point to raster tool.

The Point to Raster output matches the same values not rendered by the unique renderer.

All those datapoints have a value behind them.

Perhaps when it tries to come up with a classification scheme, a graduated renderer stops scanning a table after a certain number of records?  

From an interpolation/point to raster perspective, seems like it might be hitting this same issue and not outputting the correct result?

Point to Raster output full extent:

This does not look correct in several areas (that match up to the areas that fail to render with the graduated renderer)

[XanderBakker]

Have you tried using a Terrain Dataset? See What is a terrain dataset?—Help | ArcGIS Desktop . Try that first.

If that does not work you could also go for a TIN. See What is a TIN surface?—Help | ArcGIS Desktop  and convert that to raster.

If that does not work either, you could convert the points to text and use LAStools (see LAStools: converting, filtering, viewing, processing, and compressing LIDAR data in LAS format , check licensing, but some tools used to be free ) to convert the TXT to LAS and create a LAS dataset (Creating a LAS dataset—Help | ArcGIS Desktop ) and convert that to raster LAS Dataset To Raster—Help | ArcGIS Desktop 

For large datasets ArcGIS enterprise come with Raster Analytics Perform raster analysis—Documentation | ArcGIS Enterprise , but let's be serious 7M points is too small for that...

[XanderBakker]

And here is an "old" document on interpolation methods, however, the info is still valid:

https://www.esri.com/news/arcuser/0704/files/interpolating.pdf 

And please do convert you data into projected coordinates before interpolation and check the environment settings, since the cellsize (1 meters) might be conflicting with the geographic coordinates and yield cell sizes of 5 or 10 degrees. What is a geoprocessing environment setting?—Help | ArcGIS Desktop 

[curtvprice]

I totally second the motion on terrain -- this data structure is designed for fully specified point data like photogrammetric or lidar data. 

Xander, FYI TINs are an old ArcInfo data structure and very limited as to # of points, TIN limitations led to the development of geodatabase terrains.

[SimonJackson]

Interesting. After creating the surface, I will be doing some raster calculations against this layer and multiple other raster datasets.  Any issues using a terrain dataset for this?

[XanderBakker]

You would use Terrain To Raster—Help | ArcGIS Desktop  to create the DEM  in raster format and carry out the analysis like with any other method.