tools: generate near table - performance issues

Question

i am about to generate a near table. the input is a metric CRS and the output distance unit is set to meters, the method is set to "Geodesic" - in order to get non-projected distances. the origins are about 6.7 mio points all over Europe, near features are about 30 mio polygons. everything is spatially indext - of course. and a search radius is set to a pretty low value of 1500m ...

everything works fine, so far on sub-samples. but the performance is deeply disappointing with the 6.7mio points and definitely not up to any reasonable standard.
especially when considering that the machine is a i9-13900K, 24 core, 128 GB RAM device.
according to the windows ressourcemonitor.app exactly 3 % of the cpu performance are used to do my calculations. ... three percent !

are there any ideas for workarounds, multithread settings or anything like this to incerase performance ?
best robert

robertkalasek · Answer

didn't usse parallel processing settings up to now. thought if not set all ressources are going to be used..
no vpn no ondrive. just a superfast 2 years old machine.
the option of runnng a 1500m select sounds pretty promising -- thx for the hint !!! will try that !

i would use EPSG 3035 since the data covers whole area of europe. but this one is equal area and at the outer areas of the extent of epsg 3035 i would have reasonable problems with distance values then

so i stepped back to the geodetic-mode. since i know that postgis doesn't use spatial indices for ST_DISTANCE geodetic version, i thougth that coud be an issue in argis pro too ...

DavidPike · Answer

Possibly play with the parallel processing factor environment setting and see how the changes compare on the subset timings.

Possibly increase the spatial index levels, and data is local (not on OneDrive or making database transactions over a VPN or something!).

Possibly run a select by location of polygons within1500m and export them out, as I would guess the others are irrelevant to the analysis.  However the spatial index should really be good enough to very quickly filter these out.  However maybe worth a try.

What kind of scale distortion do you have across your extent?  I'm guessing you have something like Lambert Conformal Conic as the projection.  Wondering if you can trade a bit of accuracy for speed with a Planar computation.