topic Re: Problem with CopyRows_management to in memory with Python 3 in Python Questions

Problem with CopyRows_management to in memory with Python 3

RandyKreuziger — Sun, 12 Dec 2021 04:02:10 GMT

One of my users has a script that works in 2.7 and was working with 3.x but now breaks after the ArcGIS Pro 2.4.1 update. When copying an SDE table into memory using the arcpy.CopyRows_management all text field sizes are increasing from what there are to 5000. For example, text fields of size 1 and 100 become 5000. Our tables are a SQL 2012 / SDE 10.4.1 geodatabase. To see if I could replicate this issue I even created an empty table from scratch containing only two fields, the first is DataSource defined text length 100 and the other is Flag defined as text length 1.

Is anyone else seeing this? The user says it was working in version 3 before we pushed out Pro 2.4.1.

c:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3>python.exe h:\treg\test_inmemory2.py
SDE table is : H:\Treg\(Prod) MyDatabase@Reader.sde\MyDatabase.DBO.WS_Test
OBJECTID is a type OID with a length of 4
DataSource is a type String with a length of 100
Flag is a type String with a length of 1
Copying table H:\Treg\(Prod) MyDatabase@Reader.sde\MyDatabase.DBO.WS_Test into memory...
temp table is : memory\tmpTable
OBJECTID is a type OID with a length of 4
DataSource is a type String with a length of 5000
Flag is a type String with a length of 5000

try:
    ##Import modules
    import os, sys
    import os.path as path
    import traceback

    import arcpy
    from arcpy import env

    arcpy.env.overwriteOutput = True

    
    sdeTable = r"H:\Treg\(Prod) MyDatabase@Reader.sde\MyDatabase.DBO.WS_Test"
    tmpTable = r"memory\tmpTable"    

    print('SDE table is : {}\n' .format(sdeTable))  
    
    ##check field lengths in SDE text fields
    fields = arcpy.ListFields(sdeTable)
    for field in fields:
        print('{0} is a type {1} with a length of {2}'.format(field.name, field.type, field.length))

    print ('\nCopying table {} into memory...'.format(sdeTable))
    arcpy.CopyRows_management(sdeTable, tmpTable)    
    

    print('\ntemp table is : {}\n' .format(tmpTable))    

    ##check field length on in memory text fields
    fields = arcpy.ListFields(tmpTable)
    for field in fields:
        print('{0} is a type {1} with a length of {2}'.format(field.name, field.type, field.length))

except:
    print ('\n*** ERROR IN SCRIPT!')
    print ('\n***PYTHON ERROR INFO: \n' +  str(traceback.format_exc()))‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Problem with CopyRows_management to in memory with Python 3

JoshuaBixby — Fri, 06 Sep 2019 19:08:17 GMT

Reading over,

One of my users has a script that works in 2.7 and was working with 3.x but now breaks after the ArcGIS Pro 2.4.1 update.

The first part of the statement implies it is an ArcMap vs. ArcGIS Pro issue since ArcMap is Python 2.x and ArcGIS Pro is Python 3.x; however, the second part of the statement implies it is an ArcGIS Pro 2.4 vs Pro 2.4.1 issue.

Does the script work in Pro 2.4?

Re: Problem with CopyRows_management to in memory with Python 3

RandyKreuziger — Fri, 06 Sep 2019 19:16:11 GMT

Hi Josh,

You are correct, I didn't mention Pro vs ArcMap because we run these scripts in a CMD window. Also, we upgraded all of our ArcGIS Pros to 2.4.1 so I haven't been able to test it with an early version of Pro. I changed the script to run with python 2.7 to verify that it, at least, worked correctly in ArcMap python. The only change needed was changing line 14 from memory to in_memory. Having said that, CopyRows shouldn't be changing field definitions in the first place.

Re: Problem with CopyRows_management to in memory with Python 3

JoshuaBixby — Fri, 06 Sep 2019 19:22:17 GMT

What happens when you use the old/legacy "in_memory" in Pro instead of using the new "memory" workspace? It could be a difference in the in-memory workspace implementation more than Pro or ArcMap.

Re: Problem with CopyRows_management to in memory with Python 3

JoeBorgione — Fri, 06 Sep 2019 20:01:51 GMT

I ran a process in a Spyder window not the python window with Pro 2.4 installed on my desktop and while I don't have an answer for you, I can confirm that a text field from an egdb to in_memory stayed the same length....

Re: Problem with CopyRows_management to in memory with Python 3

RandyKreuziger — Fri, 06 Sep 2019 20:27:06 GMT

Joshua, that fixed the issue. I'll report this behavior to ESRI since it seems like a bug.

Re: Problem with CopyRows_management to in memory with Python 3

JoshuaBixby — Fri, 06 Sep 2019 21:41:32 GMT

Please mark a comment/suggestion as correct to close out the question.

Re: Problem with CopyRows_management to in memory with Python 3

DanDeegan — Sat, 07 Mar 2020 00:33:41 GMT

I am experiencing something very similar. I copy parcels to 'memory', add fields using a dictionary that comes from a template feature class. Every text/string field comes out as 5000.

edit: python 3

Re: Problem with CopyRows_management to in memory with Python 3

ToddTurner — Wed, 08 Apr 2020 00:46:23 GMT

I am experiencing the same issue in Arc Pro 2.5

I installed a fresh copy of Arc Pro 2.4 on a new laptop for some testing:

Copying into 'memory' changes all text fields to length 5000

Copy into 'in_memory' maintains the text field sizes.

Since the new capabilities of the 'memory' space are important to my project, going to 'in_memory' is not really a viable option.

Can anyone confirm that this has actually been entered as a bug at Esri, and is being worked on?

Re: Problem with CopyRows_management to in memory with Python 3

DanPatterson_Retired — Wed, 08 Apr 2020 01:35:02 GMT

If it is a bug, it isn't in the posted bug list, so maybe it hasn't been reported

https://support.esri.com/en/Search-Results#search?q=memory%20workspace&content-type=Bugs

Kory Kramer‌ 5000 sounds familiar, but not sure if it is associated with the memory workspace