memory workspace and alter Field ERROR 000664 bug.

[Anonymous User]

Updating some scripts from 2.7 to 3 in hopes of solving a memory leak that is causing an 'Out of memory' error when rebuilding a small (~50k edges) Network dataset in our maintenance tasks.  I didn't see anything mentioning this in the bug reports so thought I'd post it here if anyone else experienced this.

I started using the legacy 'in_memory' space but it looks like the CopyFeatures tool cannot output to it.  I need to create copies of data in order to modify fields and other data without changing the underlying data. I verified that nothing is written by testing with arcpy's get count and the feature count = 0. Changed to 'memory' workspace and the count is right.

The script continues up to where I need to alter a few field names.  Using the memory workspace for the outputs, the script fails with ERROR 000664: Invalid input: The type of dataset is not supported.  If I revert back to 'in_memory', it alters the field just fine.

A simple sample:

in_memory = "memory\\" # "in_memory"
env.workspace = in_memory

adrSelection = arcpy.MakeFeatureLayer_management(anyfc, 'tNR', "symbol = 319")

adr = arcpy.CopyFeatures_management(adrSelection, os.path.join(env.workspace, 'tmpLyr'))

for flds in [('st_type', 'street_type'), ('use', 'use update')]:
    arcpy.management.AlterField(adr, flds[0], flds[1], flds[1])

[JoshuaBixby · Accepted solution]

According to Alter Field (Data Management)—ArcGIS Pro | Documentation:

• When using in memory feature classes or tables, renaming the ObjectID, Shape, or other required fields such as those found in network analysis layers can result in corrupted data or unexpected behavior.

Not sure if that directly applies to your situation, but it seems close enough to point out in case you were not already aware.

UPDATE:  I just did a quick test using a basic feature class (nothing special like network analysis), and got:

arcgisscripting.ExecuteError: ERROR 000664: Invalid input: The type of dataset is not supported.
Failed to execute (AlterField).

It appears the newer memory workspace does not support altering fields.

[Anonymous User]

Thanks Josh,

I saw that in the docs, and I am not changing any of the required fields.  Just weird and a little frustrating at the same time that there seems to be issues with the in_memory and memory workspace compatibility with certain tools. I am about to apply the updates that Pro is alerting me to see if the issue persist.

[JoshuaBixby]

Historically, "in_memory" workspace was quite limited and Esri clearly stated that in-memory workspace wasn't a complete workspace, like a geodatabase.  That said, Esri has added a lot of functionality to the newer "memory" workspace, and it is close to being complete. 

Whether you are running into a defect or known limitation is unclear from the current documentation.  I suggest you open a case with Esri Support to get clarity and possibly log either a software or documentation defect.

[DanPatterson]

Good points.  There is also the project's scratch workspace and the Delete tool, for those than can manage a little extra time for writing/reading on machines with older disk hardware and/or limited memory.  Not sure where GPU is, or is going to fit into this in upcoming versions of Pro for some of the current slate of tools.  For example SA only supports a few tools but this will change

GPU processing with Spatial Analyst—ArcGIS Pro | Documentation

[Anonymous User]

Thanks Josh,

I've ran into a few limitations with the in_memory before but for the most part it was awesome for 2.7 and let us do a lot of our nightly maintenance tasks without having to temporarily output to disk. 

I finished applying the updates (now on 2.9.0) and the problem with altering the fields is still there so it looks like I'll open a case with ESRI.

[Anonymous User]

I just saw your update- thank you for checking it on your system. I converted a lot of the alter fields to be remapped fields in fieldmappings, which shaved a few minutes off the processing time for some tasks, so that is nice.  I'll have to figure out how to change it in other processes that do not involve a geoprocess that uses a fieldmap.

Side note: I think it would be extremely helpful if there were more examples of using the FieldMappings/ FieldMap objects methods.  Showing the process of finding a field by name, then changing the output field name.  Stepping through the code with a debugger, managed to do it with: 

indx = fldMap.findFieldMapIndex('name')
mapToReplace = fldMap.getFieldMap(indx)

fldMap_name = mapToReplace.outputField
fldMap_name.name = 'new_name'
fldMap_name.aliasName = 'new_name'

mapToReplace.outputField = fldMap_name

fldMap.replaceFieldMap(indx, mapToReplace)