Problem with CopyRows_management to in memory with Python 3

1581
9
Jump to solution
09-06-2019 11:43 AM
RandyKreuziger
Occasional Contributor III

One of my users has a script that works in 2.7 and was working with 3.x but now breaks after the ArcGIS Pro 2.4.1 update.  When copying an SDE table into memory using the arcpy.CopyRows_management all text field sizes are increasing from what there are to 5000.  For example, text fields of size 1 and 100 become 5000.  Our tables are a SQL 2012 / SDE 10.4.1 geodatabase.  To see if I could replicate this issue I even created an empty table from scratch containing only two fields, the first is DataSource defined text length 100 and the other is Flag defined as text length 1.

Is anyone else seeing this?  The user says it was working in version 3 before we pushed out Pro 2.4.1.

c:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3>python.exe h:\treg\test_inmemory2.py
SDE table is : H:\Treg\(Prod) MyDatabase@Reader.sde\MyDatabase.DBO.WS_Test

OBJECTID is a type OID with a length of 4

DataSource is a type String with a length of 100
Flag is a type String with a length of 1

Copying table H:\Treg\(Prod) MyDatabase@Reader.sde\MyDatabase.DBO.WS_Test into memory...

temp table is : memory\tmpTable

OBJECTID is a type OID with a length of 4
DataSource is a type String with a length of 5000
Flag is a type String with a length of 5000

try:
    ##Import modules
    import os, sys
    import os.path as path
    import traceback

    import arcpy
    from arcpy import env

    arcpy.env.overwriteOutput = True

    
    sdeTable = r"H:\Treg\(Prod) MyDatabase@Reader.sde\MyDatabase.DBO.WS_Test"
    tmpTable = r"memory\tmpTable"    

    print('SDE table is : {}\n' .format(sdeTable))  
    
    ##check field lengths in SDE text fields
    fields = arcpy.ListFields(sdeTable)
    for field in fields:
        print('{0} is a type {1} with a length of {2}'.format(field.name, field.type, field.length))

    print ('\nCopying table {} into memory...'.format(sdeTable))
    arcpy.CopyRows_management(sdeTable, tmpTable)    
    

    print('\ntemp table is : {}\n' .format(tmpTable))    

    ##check field length on in memory text fields
    fields = arcpy.ListFields(tmpTable)
    for field in fields:
        print('{0} is a type {1} with a length of {2}'.format(field.name, field.type, field.length))

except:
    print ('\n*** ERROR IN SCRIPT!')
    print ('\n***PYTHON ERROR INFO: \n' +  str(traceback.format_exc()))
0 Kudos
1 Solution

Accepted Solutions
JoshuaBixby
MVP Esteemed Contributor

What happens when you use the old/legacy "in_memory" in Pro instead of using the new "memory" workspace?  It could be a difference in the in-memory workspace implementation more than Pro or ArcMap.

View solution in original post

0 Kudos
9 Replies
JoshuaBixby
MVP Esteemed Contributor

Reading over,

One of my users has a script that works in 2.7 and was working with 3.x but now breaks after the ArcGIS Pro 2.4.1 update.

The first part of the statement implies it is an ArcMap vs. ArcGIS Pro issue since ArcMap is Python 2.x and ArcGIS Pro is Python 3.x; however, the second part of the statement implies it is an ArcGIS Pro 2.4 vs Pro 2.4.1 issue.

Does the script work in Pro 2.4?

0 Kudos
RandyKreuziger
Occasional Contributor III

Hi Josh,

  You are correct, I didn't mention Pro vs ArcMap because we run these scripts in a CMD window. Also, we upgraded all of our ArcGIS Pros to 2.4.1 so I haven't been able to test it with an early version of Pro.  I changed the script to run with python 2.7 to verify that it, at least, worked correctly in ArcMap python.  The only change needed was changing line 14 from memory to in_memory.  Having said that, CopyRows shouldn't be changing field definitions in the first place.

0 Kudos
JoshuaBixby
MVP Esteemed Contributor

What happens when you use the old/legacy "in_memory" in Pro instead of using the new "memory" workspace?  It could be a difference in the in-memory workspace implementation more than Pro or ArcMap.

0 Kudos
RandyKreuziger
Occasional Contributor III

Joshua, that fixed the issue.  I'll report this behavior to  ESRI since it seems like a bug.

0 Kudos
JoshuaBixby
MVP Esteemed Contributor

Please mark a comment/suggestion as correct to close out the question.

0 Kudos
JoeBorgione
MVP Emeritus

I ran a process in a Spyder window not the python window with Pro 2.4 installed on my desktop and while I don't have an answer for you, I can confirm that a text field from an egdb to in_memory stayed the same length....

That should just about do it....
0 Kudos
DanDeegan
New Contributor III

I am experiencing something very similar. I copy parcels to 'memory', add fields using a dictionary that comes from a template feature class. Every text/string field comes out as 5000. 

edit: python 3

0 Kudos
ToddTurner
New Contributor III

I am experiencing the same issue in Arc Pro 2.5

I installed a fresh copy of Arc Pro 2.4 on a new laptop for some testing:

   Copying into 'memory' changes all text fields to length 5000

   Copy into 'in_memory' maintains the text field sizes.

Since the new capabilities of the 'memory' space are important to my project, going to 'in_memory' is not really a viable option.

Can anyone confirm that this has actually been entered as a bug at Esri, and is being worked on?

0 Kudos
DanPatterson_Retired
MVP Emeritus

If it is a bug, it isn't in the posted bug list, so maybe it hasn't been reported

https://support.esri.com/en/Search-Results#search?q=memory%20workspace&content-type=Bugs

Kory Kramer‌  5000 sounds familiar, but not sure if it is associated with the memory workspace

0 Kudos