Memory issue inserting a large dataset in GDB with ArcObjects

KevinLECOCQ — Sat, 11 Dec 2021 22:23:46 GMT

I am using the ArcObjects API for Java to load a very large dataset in a File Geodatabase in one feature class.

The whole dataset I have to load is composed of 200 datasets of 30k polygons, and I load the datasets one after the other. I read data from XML files (I am sure there is no memory leak in this step).

The loading process is in a BaseGeoprocessingTool.

Here is my code to write the data of one dataset in the feature class:

 IFeatureClass featureClass = ...
    IFeatureCursor insertCursor = featureClass.IFeatureClass_insert(false); // true uses too much memory
    IFeatureBuffer featureBuffer = featureClass.createFeatureBuffer();
    IGeometry geometry = null;
    List<Integer> fieldIdsToClear = new ArrayList<Integer>();
    int i = 0;

    for(FeatureBean featureBean : features) {
        if(log.canContinue()) {
            geometry = featureBean.geometry.getGeometry();
            geometry.setSpatialReferenceByRef(fromSpatialReference);
            if(toBeProjected) geometry.project(toSpatialReference);
            featureBuffer.setShapeByRef(geometry);

            // reset the fields in the featureBuffer
            for(Integer fieldId : fieldIdsToClear) {
                featureBuffer.setValue(fieldId, null);
            }
            fieldIdsToClear.clear();

            featureBuffer.setValue(idFieldId, (int)featureBean.id);

            for(Entry<String, String> tag : featureBean.tagMap.entrySet()) {
                int fieldId = fieldIdMap.get(tag.getKey());
                featureBuffer.setValue(fieldId, tag.getValue());
                fieldIdsToClear.add(fieldId);
            }

            insertCursor.insertFeature(featureBuffer);

            i++;
            if(i > 1000) {
               insertCursor.flush();
               i = 0;
            }
        }
    }

    insertCursor.flush();

    Cleaner.release(geometry);
    Cleaner.release(featureBuffer);
    Cleaner.release(insertCursor);
    Cleaner.release(featureClass);

After reading these advices, I use an insert cursor instead of IFeature.store() so that process is ~5x faster. So, I do not recreate the FeatureBuffer each time I insert a new feature. Instead of, I reuse it and clear the necessary fields.

This process works well and is very fast. Nevertheless, there is a memory leak. Watching ArcMap.exeprocess memory, we can see that it uses about 100 Mb to load each dataset, then the memory is not released, even if objects are released in my code. When memory is about 1300 Mb (only after loaded the 5th dataset), the process crashes on the exception AutomationException: One or more arguments are invalid

I have tried different Java max heap sizes with JavaConfigTool.exe between 512 and 1900 Mb, but it doesn't change anything.

What solution could I try to be sure that ArcGIS really release useless objects? As I have a lot of datasets to load, I really need to release to get back the 100 Mb each time a dataset has been loaded.

I also tried to flush every 1, 1000 or 2000 features but it doesn't change anything in memory usage.

Any help is welcome to optimize my process.

I use ArcGIS 10.2 with Background Geoprocessing (64bits) and have 8 Go of RAM.

Re: Memory issue inserting a large dataset in GDB with ArcObjects

DuncanHornby — Fri, 08 May 2015 09:22:54 GMT

I do not program in java so not sure if this is even an option for you. There is an interface called IFeatureClassLoad which alters the behaviour of the Feature class when doing bulk loads, have a look at that?

Re: Memory issue inserting a large dataset in GDB with ArcObjects

KevinLECOCQ — Mon, 18 May 2015 15:56:02 GMT

Yes, I tried this interface but I don't know why, when I cast my feature class to IFeatureClassLoad, I get a cast exception : java.lang.ClassCastException: com.esri.arcgis.geodatabase.IFeatureClassProxy cannot be cast to com.esri.arcgis.geodatabase.IFeatureClassLoad

However, according to the documentation, IFeatureClassLoad is implemented for File GDB feature classes.

topic Re: Memory issue inserting a large dataset in GDB with ArcObjects in ArcObjects SDK Questions

Memory issue inserting a large dataset in GDB with ArcObjects

Re: Memory issue inserting a large dataset in GDB with ArcObjects

Re: Memory issue inserting a large dataset in GDB with ArcObjects