I am using the ArcObjects API for Java to load a very large dataset in a File Geodatabase in one feature class.
The whole dataset I have to load is composed of 200 datasets of 30k polygons, and I load the datasets one after the other. I read data from XML files (I am sure there is no memory leak in this step).
The loading process is in a BaseGeoprocessingTool.
Here is my code to write the data of one dataset in the feature class:
IFeatureClass featureClass = ...
IFeatureCursor insertCursor = featureClass.IFeatureClass_insert(false); // true uses too much memory
IFeatureBuffer featureBuffer = featureClass.createFeatureBuffer();
IGeometry geometry = null;
List<Integer> fieldIdsToClear = new ArrayList<Integer>();
int i = 0;
for(FeatureBean featureBean : features) {
if(log.canContinue()) {
geometry = featureBean.geometry.getGeometry();
geometry.setSpatialReferenceByRef(fromSpatialReference);
if(toBeProjected) geometry.project(toSpatialReference);
featureBuffer.setShapeByRef(geometry);
// reset the fields in the featureBuffer
for(Integer fieldId : fieldIdsToClear) {
featureBuffer.setValue(fieldId, null);
}
fieldIdsToClear.clear();
featureBuffer.setValue(idFieldId, (int)featureBean.id);
for(Entry<String, String> tag : featureBean.tagMap.entrySet()) {
int fieldId = fieldIdMap.get(tag.getKey());
featureBuffer.setValue(fieldId, tag.getValue());
fieldIdsToClear.add(fieldId);
}
insertCursor.insertFeature(featureBuffer);
i++;
if(i > 1000) {
insertCursor.flush();
i = 0;
}
}
}
insertCursor.flush();
Cleaner.release(geometry);
Cleaner.release(featureBuffer);
Cleaner.release(insertCursor);
Cleaner.release(featureClass);After reading these advices, I use an insert cursor instead of IFeature.store() so that process is ~5x faster. So, I do not recreate the FeatureBuffer each time I insert a new feature. Instead of, I reuse it and clear the necessary fields.
This process works well and is very fast. Nevertheless, there is a memory leak. Watching ArcMap.exeprocess memory, we can see that it uses about 100 Mb to load each dataset, then the memory is not released, even if objects are released in my code. When memory is about 1300 Mb (only after loaded the 5th dataset), the process crashes on the exception AutomationException: One or more arguments are invalid
I have tried different Java max heap sizes with JavaConfigTool.exe between 512 and 1900 Mb, but it doesn't change anything.
What solution could I try to be sure that ArcGIS really release useless objects? As I have a lot of datasets to load, I really need to release to get back the 100 Mb each time a dataset has been loaded.
I also tried to flush every 1, 1000 or 2000 features but it doesn't change anything in memory usage.
Any help is welcome to optimize my process.
I use ArcGIS 10.2 with Background Geoprocessing (64bits) and have 8 Go of RAM.
