Processing cursor is taking long time

GhassanKarwchan · ‎03-09-2013

I am maintaining a code on arcobjects 10.1, and there is a bug happening.
I have a code that is parsing through a cursor, and generate a where clause statement to run it on a second query.
So the code is like this


IFeatureCursor containsCursor = null;
IFeatureClass featureClass = null;
ISpatialFilter spatialFilter = new SpatialFilterClass();
IFeature tempFeature = null;

try
 {
  featureClass = workspace.OpenFeatureClass("tablename");
  spatialFilter.Geometry = operationalUnit.Shape;
  spatialFilter.GeometryField = featureClass.ShapeFieldName;
  spatialFilter.SpatialRel = esriSpatialRelEnum.esriSpatialRelContains;

  containsCursor = featureClass.Search(spatialFilter, true);
  tempFeature = containsCursor.NextFeature();
  String whereClause = null;
  String fieldName = tempFeature.Table.OIDFieldName;
  while (tempFeature != null)
  {
   if (tempFeature.HasOID)
   {
    whereClause = whereClause + String.Format(" and {0} <> {1}", tempFeature.Table.OIDFieldName, tempFeature.OID);
   }

   tempFeature = containsCursor.NextFeature();
  }                

  (do something with the where clause)
 }
 finally
 {
  Helper.ReleaseObject(featureClass);
  Helper.ReleaseObject(tempFeature);
  Helper.ReleaseObject(containsCursor);
  Helper.ReleaseObject(spatialFilter);
 }

The loop that is generating the where clause is talking long time.
and when it is done, it tries to connect to the database, and the connection is dead.

Now, the funny thing is the slowness in only on the Dev server.
When I run it on my machine is fast.

This is my full details information

On my Dev server: It is a virtual machine, with Windows 2008R2
I run ArcGIS serer 10.1
I run the process binding to the Server, and to the server license.
I compile the process to run as 64 bit in order to bind to the server.

I access the database through Application Server SDE 9.3

P.S: I don't use direct connection (and we are not going to change the code to use it)

I measured the length of the statement, and on the server for 3500 records is taking > 45 seconds.

On my machine: my machine is Windows 7 with 8 gig Ram.
I run ArcGIS Desktop.
I compile the code to x86, and run binding to the desktop.

For the same amount of records it is taking < 4 seconds.

What is wrong there?

JasonPike · ‎03-11-2013

Server and Workstation modes of the Garbage Collector algorithm are slightly different, which may account for some or all of the differences between your development machine and the server machine.
http://msdn.microsoft.com/en-us/library/bb680014.aspx

One thing I see that will speed things up is to use the StringBuilder class to accumulate strings for your where clause.
http://www.dotnetperls.com/stringbuilder

Also, you're not releasing things within the loop, so a lot of unmanaged memory is being held onto longer than necessary. I put comments in these places for your consideration.

IFeatureCursor containsCursor = null;
IFeatureClass featureClass = null;
ISpatialFilter spatialFilter = new SpatialFilterClass();
IFeature tempFeature = null;

try
{
 featureClass = workspace.OpenFeatureClass("tablename");
 spatialFilter.Geometry = operationalUnit.Shape;
 spatialFilter.GeometryField = featureClass.ShapeFieldName;
 spatialFilter.SpatialRel = esriSpatialRelEnum.esriSpatialRelContains;

 containsCursor = featureClass.Search(spatialFilter, true);
 tempFeature = containsCursor.NextFeature();

 StringBuilder whereClauseBuilder = new StringBuilder();
 String fieldName = tempFeature.Table.OIDFieldName;;
 
 while (tempFeature != null)
 {
  if (tempFeature.HasOID)
  {
   // put tempFeature.Table in a variable and release here so that you release each instance in the loop
   whereClauseBuilder.Append( String.Format(" and {0} <> {1}", tempFeature.Table.OIDFieldName, tempFeature.OID ));
  }
  
  // release 'tempFeature' here so that you release each instance in the loop
  tempFeature = containsCursor.NextFeature();
 }                

 String whereClause = whereClauseBuilder.ToString();
 // do something with the where clause
}
finally
{
 Helper.ReleaseObject(featureClass);
 Helper.ReleaseObject(tempFeature);
 Helper.ReleaseObject(containsCursor);
 Helper.ReleaseObject(spatialFilter);
}

AlexanderGray · ‎03-11-2013

I compile the process to run as 64 bit in order to bind to the server.

ArcObjects is 32 bits, are you running this inside an ArcGIS server SOC?

Jason has good points but I don't see them accounting for that much difference.
I would actually build a list of strings, at the end use the toArray in a string.joing with "and" and a separator.
http://msdn.microsoft.com/en-us/library/57a79xd0.aspx
Avoids having to deal with trailing or leading "and"

Check how long it takes to get the first feature, that is usually when the query is executed on the database side. The creation of the cursor usually just validates the query but doesn't perform it. If the return of the first feature is slow, then it is slow on the database.

It is hard to say what is the best way without knowing what you plan on doing with it. The whereclause looks like you are building a way to exclude these features from something else. I am not sure you need to make this query in the first place, there are limits to length of the where clauses you can pass in the query filter. If you absolutely need a cursor, you can limit the fields in the first spatial filter (Shape and ObjectID.) However, if you make a selection instead, the selection is an array of objectids (ISelectionSet.IDs.) You can use this array to make your query filter for the table, or better yet create a selection on the table and use removeList from the selectionset and the you can call search on the selectionset and get a cursor off of the selection. ISelectionSet2 even lets you get an update cursor off of the selectionset. You can pass in additional conditions in those cursors.

GhassanKarwchan · ‎03-11-2013

ArcObjects is 32 bits, are you running this inside an ArcGIS server SOC?

No , I am not running inside SOC.
I am just running it inside a window service application.
So, it is similar to running any command line application, but it is 64, and binding to the server.

Is that a valid configuration?

These are the requirements of the client, and I had to go with it.

The code is already coming from 9.3 code, that was running as 32 bit and running under Engine license.
The client asked to converted to run with 10.1, and run on Arcgis server , and bind to server license (so it has to be 64 bit).

GhassanKarwchan · ‎03-11-2013

ArcObjects is 32 bits, are you running this inside an ArcGIS server SOC?

Check how long it takes to get the first feature, that is usually when the query is executed on the database side. The creation of the cursor usually just validates the query but doesn't perform it. If the return of the first feature is slow, then it is slow on the database.

It takes only part of a second to run the query and get the first feature.
I put log inside the where clause, and actually to go through 3 thousands records was taking sometimes 3 minutes.
It was processing less than 10 per second.

That on 64 bit, binding to the server.
But when I run it on 32 bit, the whole 3000 takes less than 5 seconds.
and again that time is during processing the loop, so it is after getting the first feature(which is running the query).

GhassanKarwchan · ‎03-11-2013

Server and Workstation modes of the Garbage Collector algorithm are slightly different, which may account for some or all of the differences between your development machine and the server machine.
http://msdn.microsoft.com/en-us/library/bb680014.aspx

Those are very valid points.
Thanks ScJPike

I will try them.
As for the string builder, I already did that, and didn't help so much.

JasonPike · ‎03-11-2013

Those are very valid points.
Thanks ScJPike

I will try them.
As for the string builder, I already did that, and didn't help so much.

No problem. I'm really disappointed that StringBuilder didn't buy you much. I'd like to take one more crack at it. The implementation I suggest below makes some assumptions (like the OID field name and Table being the same), but if those assumptions are OK, it should save you a great deal in garbage collections.

Good luck!

IFeatureCursor containsCursor = null;
IFeatureClass featureClass = null;
ISpatialFilter spatialFilter = new SpatialFilterClass();
IFeature tempFeature = null;
ITable table = null;

try
{
 featureClass = workspace.OpenFeatureClass("tablename");
 spatialFilter.Geometry = operationalUnit.Shape;
 spatialFilter.GeometryField = featureClass.ShapeFieldName;
 spatialFilter.SpatialRel = esriSpatialRelEnum.esriSpatialRelContains;

 containsCursor = featureClass.Search(spatialFilter, true);
 tempFeature = containsCursor.NextFeature();
 table = tempFeature.Table;
 String fieldName = table.OIDFieldName;
 String andOIDFieldNotEqual = " and " + fieldName + " <> ";
        StringBuilder whereClauseBuilder = new StringBuilder();

 while (tempFeature != null)
 {
  if (tempFeature.HasOID)
  {
   whereClauseBuilder.Append(andOIDFieldNotEqual);
   whereClauseBuilder.Append(tempFeature.OID);
  }

  Helper.ReleaseObject(tempFeature);
  tempFeature = containsCursor.NextFeature();
 }                

 String whereClause = whereClauseBuilder.ToString();
 (do something with the where clause)
}
finally
{
        Helper.ReleaseObject(table);
 Helper.ReleaseObject(featureClass);
 Helper.ReleaseObject(tempFeature);
 Helper.ReleaseObject(containsCursor);
 Helper.ReleaseObject(spatialFilter);
}

LeoDonahue · ‎03-11-2013

Releasing a reference to tempFeature inside the while loop? How would the while loop continue?

JasonPike · ‎03-12-2013

Releasing a reference to tempFeature inside the while loop? How would the while loop continue?

Good catch, Leo. I got in a hurry and pasted the release after the assignment of the next feature instead of before it.

This is what I intended, which was to release the previously used feature before assigning the new one.

Helper.ReleaseObject(tempFeature);
tempFeature = containsCursor.NextFeature();

I'll fix the original as well.

FabioPiona_de_Sousa · ‎06-06-2016

Great solution and works fine for me. I have an Console Application using ArcObjects SDK 10.3.1 thats run in a Windows 64bits Server and didn´t work after some loops, for example, it reads 20 features and did not anything more.

Regards

Fabio