topic Re: Spatial Join CRASHING at 10 in Geoprocessing Questions

Spatial Join CRASHING at 10

TedCronin — Fri, 23 Jul 2010 00:58:57 GMT

So, I can run about 5 consecutive processes for a spatial join before Wing crashes and processes no more. Using a Point and a Poly layer to Join. Point layer is 774,796 records. Poly layers vary, but are much smaller than point. Spatial Join in a list, which makes this a tedious process in that I get to wait for the crash and then remove the layer from the list and then reprocess the list, so it kinda makes using lists not optimal. Physical Memory drops to about 34,000 K before the crash. It steadily goes down from the first to the sixth layer, with the sixth layer not completing successfully.

Good Times.

oh, this is running in a File GDB.

There is NO error number generated, it just stops. Believe it is runaway memory.

Re: Spatial Join CRASHING at 10

NobbirAhmed — Wed, 04 Aug 2010 16:06:41 GMT

I would like to repro the issue. In Wing, are you using the mapping module to get the layers? Could you post a snapshot of your code? Thanks.

Nobbir Ahmed
Geoprocessing & Analysis, ESRI.

Re: Spatial Join CRASHING at 10

ChrisSnyder — Wed, 04 Aug 2010 16:25:40 GMT

Sounds like the classic gp memory leak issue...

Unless ESRI fixes it, only workaround is to break up the process into pieces using os.spwanv, subprocess, etc.

Re: Spatial Join CRASHING at 10

TedCronin — Fri, 06 Aug 2010 00:44:21 GMT

@NobbirAhmed

Will do next week, but I need to coordinate the data transfer, so email me at tcronin@asrclkrec.com

@chris.snyder@wadnr.gov
Perhaps, I get to use more of subprocess, which I like 🙂

Re: Spatial Join CRASHING at 10

JamesTanis — Thu, 09 Sep 2010 13:32:08 GMT

I am having the same issue. I created a model that does a spatial join around 6 to 8 times and it crashes everytime now in 10. The model works fine in 9.3.1 on a different computer with the same dataset.

I even tried making the dataset smaller to run through the model in 10 and it still crashes. One thing I noticed was I tried running each of these spaital joins seperatly in 10 through tool box in arc catalog and after each spatial join the memory never gets released so after my first sptial join my memory usage for arc catalog was at 303,300k and after the second one it was at 660,450k and the thrid one it was at 943,000k, and the next spatial join crashed.

Re: Spatial Join CRASHING at 10

ChrisSnyder — Thu, 09 Sep 2010 15:05:48 GMT

Classic gp memory leak...

Hope that all the progress that ESRI made on this issue doesn't resurface in v10...

Re: Spatial Join CRASHING at 10

NobbirAhmed — Thu, 28 Apr 2011 15:17:45 GMT

The memory leak is fixed in 10.0 sp2.

Re: Spatial Join CRASHING at 10

TedCronin — Thu, 28 Apr 2011 18:54:26 GMT

Good news, thanks for the update.

Re: Spatial Join CRASHING at 10

SowjanyaSunkara — Fri, 08 Jul 2011 17:23:41 GMT

Is there any fix/workaround to this issue in ArcGIS 10 SP1?
We have a model that works fine in 9.3.1 but the Spatial Join tool crashes in ArcGIS 10 Sp1.

My solution...

DuncanHornby — Mon, 21 Nov 2011 15:19:42 GMT

Just want to add my "2 cents" to this thread. I have spent the last 2 hours trying to run a spatial join of points to points using the closest match option. Every time it crashed. I tried moving the data to a file geodatabase, to shapefiles, stripped down the fields to the bare minimum, removed domains, you name it I did it.

I then asked someone in my office if they could run it to rule out this stupid roaming profile setup we have on our network. Hey presto it worked! AHHH! Anyway the only difference I noticed was that her geoprocessing tool ran as back ground task. I had turned that off along time ago as I think it's pointless and I find the dialog helpful to watch whats going on. So I out of desperation I turned this back on and it worked.

Seriously cheesed off about this...

Re: Spatial Join CRASHING at 10

NobbirAhmed — Mon, 21 Nov 2011 17:17:46 GMT

@Hornbydd

Hi Duncan, thanks a lot for describing the issue elaborately. I'll try to repro it locall and submit a bug if necessary.

Re: Spatial Join CRASHING at 10

BrianBatten — Thu, 09 Feb 2012 15:02:50 GMT

Last post mentioned this was fixed in 10 SP2,
I'm running SP3 and experiencing a similar issue. A model I've created that uses Spatial Join works fine in 9.3.1 but crashes in Arc10. I isolated it down to just the Spatial Join command, and created a model that just executes spatial join and it crashes every time. I tested the background geoprocessing on/off solution with no success. Same process works fine in 10 when executed outside of ModelBuilder.

The spatial join is one-to-many, closest with a set maximum search distance.

Any advice on solutions would be most appreciated.

Re: Spatial Join CRASHING at 10

KeithAdams — Tue, 28 Feb 2012 13:46:38 GMT

Hi all-

We have now stumbled onto the same land mine after (finally) completing our migration from 9.3.1 to 10.0 SP 3. Our situation is:

We have a series of Python-based GP scripts that get launched from inside Informatica workflows. The spatial join step (joining county data to point features, where the size of the point feature class ranges from a few dozen to about 10,000 features) fails and throws an o/s level exception. When the same code is run from a Windows CMD shell, we have no problem; it only happens when executed from within the Informatica workflows.

Our environment is:

Windows Server 2003 with all the latest and greatest service packs, KB patches, etc.

ArcGIS 10.0 SP 3

MS SQL Server 2005 plus ArcSDE

Informatica 8.1.1 (which Informatica no longer supports- that's next up on the modernization list)

Data are being processed from a combination of file GDB (point features) and ArcSDE layers (counties)

Text of the Windows Application event log is as follows:

[INDENT]Event Type: Error
Event Source: .NET Runtime 2.0 Error Reporting
Event Category: None
Event ID: 1000
Date: 2/25/2012
Time: 1:27:48 PM
User: N/A
Computer: XXXXXXX
Description:
Faulting application python.exe, version 0.0.0.0, stamp 4ba3e443, faulting module atio2kad.dll, version 4.0.0.1119, stamp 3bcc6880, debug? 0, fault address 0x000d0d49.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
[/INDENT]

Thanks for any help / insights you have that fixes this problem. Our data managers will greatly appreciate not having to start their work day at 2 a.m. every day!

Last post mentioned this was fixed in 10 SP2,
I'm running SP3 and experiencing a similar issue. A model I've created that uses Spatial Join works fine in 9.3.1 but crashes in Arc10. I isolated it down to just the Spatial Join command, and created a model that just executes spatial join and it crashes every time. I tested the background geoprocessing on/off solution with no success. Same process works fine in 10 when executed outside of ModelBuilder.

The spatial join is one-to-many, closest with a set maximum search distance.

Any advice on solutions would be most appreciated.

Re: Spatial Join CRASHING at 10

NobbirAhmed — Tue, 28 Feb 2012 20:48:05 GMT

Last post mentioned this was fixed in 10 SP2,
.......
The spatial join is one-to-many, closest with a set maximum search distance.

Just tried your scenario (spatial join in MB) on 10 sp3 - does not crash on my machine. I can further investigate if you can provide the data + model.

Re: Spatial Join CRASHING at 10

AaronBarkhurst — Sat, 03 Mar 2012 19:35:49 GMT

Last post mentioned this was fixed in 10 SP2,
I'm running SP3 and experiencing a similar issue. A model I've created that uses Spatial Join works fine in 9.3.1 but crashes in Arc10. I isolated it down to just the Spatial Join command, and created a model that just executes spatial join and it crashes every time. I tested the background geoprocessing on/off solution with no success. Same process works fine in 10 when executed outside of ModelBuilder.

The spatial join is one-to-many, closest with a set maximum search distance.

Any advice on solutions would be most appreciated.

I am having the same problem with spatial join in a python script. Setting the match_option parameter to "CLOSEST" is the only thing that crashes the script. I absolutely need that parameter as "CLOSEST" and am not crazy about having to write the code to loop through the geometries to accomplish that without the Spatial Join tool. Has ANYONE found a cause/solution to this, or is it another overseen esri bug that allows tools to be used as stand-alone from the desktop applications only????

Re: Spatial Join CRASHING at 10

NathanJennings — Mon, 12 Mar 2012 18:57:16 GMT

I am getting a problem with the spatial join using Arc10.0, SP4 using ArcSDE 9.2 SP6 feature classes using ~150,000 points overlaid with a polygon file of ~18 polygons, both SDE feature classes. All of the records appear, but all but the first row of attributes are "null". In the first row all of the input fields are populated, but the spatial join field(s) are null, too. Output is SDE feature class.

Running from OS is Windows 7 Pro, 32-bit, SP-1 accessing an Oracle 10g relational database.

I selected about 20000 records interactively in ArcMap of the point feature class, then ran the spatial join and the data was populated as expected for the selected records. The output was only of the selected point features.

Was running fine on Arc10 SP2 with the full data set.

Ran successfully with the full data set on Arc 9.3.1 using the same ArcSDE 9.2 SP6 data.

Ran successfully with full data set in Arc10 SP 3 using the same ArcSDE 9.2 SP6 data.

Ran approximately 12000 out of the 150,000 of the records were processed. It appears the process broke when the spatial join encountered a null value in the polygon layer for the shapefile data. This was not the case for SDE feature classes.

I have a Python script that performs a number of Spatial Joins that I perform the Spatial Joins as well as other routines using SDE data. Scripts and ArcGIS Toolbox routines run successfully to completion, but produce "null" output as described above. We have typically run these scripts over the last 4 years on a monthly basis.

The "Name" field is the only field that is added as part of the Spatial Join operation, in this case. See the attached document.

I would need to FTP or send data via a different method to provide data.

Thank you,
Nate

Re: Spatial Join CRASHING at 10

DanMcCoy — Fri, 30 Mar 2012 20:51:41 GMT

The problem Nate is experiencing was logged as NIM079126:
http://support.esri.com/en/bugs/nimbus/TklNMDc5MTI2

However, it looks like maybe it was listed as "declined" and that maybe ESRI was not able to reproduce it. This particular bug only seems to affect the �??Spatial Join�?� geoprocessing task (used by ArcToolbox, ModelBuilder, and Python), but not the �??Join data from another layer based on spatial location�?� option of the join data dialog available from clicking a layer in the table of contents (TOC)->joins & relates->join�?�

Also, I believe that you will only experience the bug with the geoprocessing task, if any of the �??target features�?� do not have a spatial relationship to any �??join features�?�. For example, if you are trying a point-in-poly intersect, any points that are not within a polygon could throw off the results of the entire operation. You can avoid this by unchecking �??Keep All Target Features�?� in the tool dialog or by using the KEEP_COMMON join_type option in your code.

If anyone else is running into this same problem after installing SP4, please contact ESRI Support to ensure that they are working to get this fixed.

Thanks,

Dan

Re: Spatial Join CRASHING at 10

PamTurner — Mon, 02 Apr 2012 02:51:52 GMT

Same here. Model Builder process crashes when it hits the "Spatial Join" task. Problem appears to be with the "closest" option. I need to use this option because of the licensing level on our server. Anyone have a work around? I'd like to use "Near" or one of the simplier processing tools, but no dice. Using ArcGIS 10.0 SP4.

I'll try the bug route with ESRI.

Re: Spatial Join CRASHING at 10

RussCoffey — Wed, 04 Apr 2012 20:07:49 GMT

Using ArcGIS 10 sp4, performing a polygon/points one-to-one spatial join, I am experiencing the same issue described where only the first record has any of the field mappings applied property. All remaining records are null on the joined fields (although the "Join_Count" field is correctly populated for all rows).

I am seeing the same behaviour whether I run the tool manually in ArcMap or as code in a stand-alone Python script. Also, storage format (SDE, File GDB, shapefile) doesn't seem to matter.
I saw the bug report mentioned above, and status of "Declined" sounds like it's not going to be fixed, so I'll submit a bug report as well.

However, unchecking the "Keep All Target Features" option was successful as a work-around for me in all cases I tried, so thanks to Dan for that suggestion!

Re: Spatial Join CRASHING at 10

DanMcCoy — Mon, 23 Apr 2012 15:24:27 GMT

The problem Nate is experiencing was logged as NIM079126:
http://support.esri.com/en/bugs/nimbus/TklNMDc5MTI2

However, it looks like maybe it was listed as "declined" and that maybe ESRI was not able to reproduce it....

FYI - ESRI has resurrected this bug and will be soon releasing a patch...

Bug: The Spatial Join tool produces NULL results for some fields/features on ArcGIS 10 SP4
http://support.esri.com/en/knowledgebase/techarticles/detail/40174