Link fire hotspot point data to monthly meteorological data (closest to fire date)

[CraigWayson1]

Hi - I hope someone can point me in the right direction. Problem is as follows:

I have a point feature layer of fire hotspots (NASA MODIS product) that has a dateonly attribute (among others) for the past 25 years or so, subset to my AOI, etc. About 780,000 features. 

I have a subset of NASA POWER monthly meteorological data, subset to AOI, variables and matching date range of fire data.

I need to add to the fire features' attribute table the corresponding meteorological values of monthly surface temperature, precipitation, etc. 

If I try to create a table using the fire locations as the sample layer using the multidimensional tool, 'Sample', for all fires and all met monthly data across the years, the computational load is too high (I have had the process running on a pretty fast machine for the last 50 hours or so and still not done). Obviously, if I use the 'Sample' tool and limit the time dimension to 1 year, the table pops out pretty quickly. As it stands, I am asking Pro to make a table with about 7 billion elements. Not needed for what I want, but I was going to just join the desired information from that master table to my point feature layer for use later in a MaxENT or Random Forest model.

There must be an easier way. How can I search through the met data spatially- and time-wise and grab what I need? I'd like to stay in ArcGIS Pro as I am trying to brush up on my ArcGIS Pro skills, but if the answer is 'do it in Python', I'll just use Matlab or R as my Python is pretty bad.

Thanks!

[DavidPike]

The first things I'd check would be:

1.  A Spatial index created/existing on both layers to the highest level possible first.

2.  Attribute indexes on your date fields you're matching against.

3.  Data location - ensure this is done locally, if you're doing this over VPN, sde, database, OneDrive, Fileshare etc etc. this is the problem.  Process it locally.

4.  Parallel Processing Factor (Environment setting)—ArcGIS Pro | Documentation  Check if you can use it with your tool, and use it to the highest level if so.

[CraigWayson1]

Thanks for the tips. That sped things up and the process finishes. However, I am now confronted with a new issue/bug. The output table from 'Sample' caps at 276,000 rows. I've tried different point feature inputs of varying lengths and always the output table caps at 276,000, unless the point feature had so few points as to not produce an output greater than 276,000 rows. I tried passing the CONFIG keyword to increase the file size (though in no way was I going to hit the normal 1 TB limit). I am very much at a loss on where to go from here. I am running v3.3.4. 

[DavidPike]

May be obvious but I'll say it anyway - I'd ensure the output location is a Geodatabase instead of a Shapefile (10MB limit). 

I'm not really familiar with this tool, but it might be possible it's skipping some features where it can't find a closest match in space or time.  Something wrong with the date attribute or NoData for the raster etc. (also be aware of resampling) Or possibly the feature itself.  Check Geometry and Repair geometry tool might be worth a try also.  Projections / Coordinate systems of the inputs.

If it was me I'd identify a few features that are missing from your table and overlay them onto the raster at those times. It might help you rule out some possibilities.

[CraigWayson1]

Thanks for the quick answer. Input point features and output Table all go to a Geodatabase. Geometries, projections etc. all up to snuff. Odd thing is that it does not skip over features. When Pro draws the points on the screen it always starts from the same place and fills in along the same pattern. What I have found is that the output table, which keeps the met data and X, Y coords, when I add X, Y to the Map from the table, it always starts drawing from the same area and runs until it hits 276,000 points. It is almost as if the Sample tool cursors across the point features and then stops at 276,000 features. When I subset the point features by Year, fires that occurred in any given year are all across my AOI, but it only processes, from the same place, until it hits that cap. Weird.

I am doing a reinstall/patch to move up to 3.4.3 and will try again.