Change from ArcGIS Pro 2.6.3 to 2.7.1 causes error in arcpy GP Tool

[cmwrenn3]

Hey, everyone.

Has anyone installing Pro 2.7.1 (Python 3.7.9) run into an issue using scripts written in Python 3.6.10 (Pro 2.6.3), specifically Geoprocessing Tools?

We have a script, written in 3.6.10 that calculates three fields, and when run from a machine with Pro 2.6.3, the calculations are correct.  When run from a machine with 3.7.9/Pro 2.7.1, one of the fields calculates correctly (current date & time), while the others don't (copy the number of COVID cases from a joined table; divide the number of cases by population, then multiply by 100,000).  The first one places a Null into the COVID cases field which causes the second to become Null.

Thanks,

Charlie  

[DanPatterson]

The script would help.  There is nothing in python that would have caused and error, so that leaves something in arcpy or calls to tools that would have changed.  

If it is the Calculate Field (Data Management)—ArcGIS Pro | Documentation

tool that you are using, you can see if there are syntax changes between Pro 2.6 and 2.7 in the toggle beneath the title

[cmwrenn3]

Here's the script (slightly reformatted for display here.

I previously noticed the syntax changes between 2.6.3 and 2.7  (though Esri's sample scripts are the same for both versions), and have tried both with the same results.  Again, the first one fails with a null result; the second may or may not be failing (since the first provides a null value that the second references), and the third (date) works on both python versions.

# Update COVID19 Zip Code data
def email_results(results):
"""Email the completion results."""
msg = EmailMessage()
msg.set_content(results)
msg['Subject'] = 'COVID19 ZipCode Update Completed'
msg['From'] = 'cmwrenn@co.pg.md.us'
msg['To'] = 'cmwrenn@co.pg.md.us'
server = smtplib.SMTP('smtp.co.pg.md.us', 25)
server.starttls()
server.send_message(msg)
server.quit()
return

import arcpy
from arcpy import mp
import smtplib
from email.message import EmailMessage

aprx = mp.ArcGISProject(r'\\ ... \ZipCodeRate.aprx') # Production
#aprx = mp.ArcGISProject(r'C:\Users\ ... \ZipCodeRate.aprx') # Testing
m = aprx.listMaps('Layers')[0]
zipcovid = m.listLayers('*COVID_STATEZIP')[0]
zipmd = m.listLayers('ZIPCodes_MD')[0]
#arcpy.AddMessage('Calculating CASES')
print('Calculating CASES') ## Flow Trace

# Pro 2.6.3/Py 3.6.10
arcpy.CalculateField_management(zipcovid, 
                                'CASES',
                                '!L0ZIPCODES_MD.ProtectedCount!')
# Pro 2.7.1/Py 3.7.9
#arcpy.management.CalculateField(zipcovid,
#                                'CASES',
#                                '!L0ZIPCODES_MD.ProtectedCount!')

#arcpy.AddMessage('Calculating RATE')
print('Calculating RATE') ## Flow Trace

# Pro 2.6.3/Py 3.6.10
arcpy.CalculateField_management(zipcovid,
                                'RATE',
                                '!CASES! / !POP100! * 100000')

# Pro 2.7.1/Py 3.7.9
#arcpy.management.CalculateField(zipcovid,
#                                'RATE',
#                                '!CASES! / !POP100! * 100000')

print('Calculating RIMPRT_DATE') ## Flow Trace

# Pro 2.6.3/Py 3.6.10
arcpy.CalculateField_management(zipcovid,
                                'IMPRT_DATE',
                                'time.strftime("%m/%d/%Y")')

# Pro 2.7.1/Py 3.7.9
#arcpy.management.CalculateField(zipcovid,
#                                'IMPRT_DATE',
#                                'time.strftime("%m/%d/%Y")')

print('Saving project file') ## Flow Trace

aprx.save()

results = 'The ZipCodes have been updated with the current Cases and Population per 100,000 cases.'
email_results(results)

print('Completed') ## Flow Trace

[Anonymous User]

Could you reference the layer in its location without having to open the aprx and step through the project elements? 

Maybe it is the ArcGISProject module is causing the issue.  (There are a few other topics posted recently that are using the ArcGISProject module and they too seem to be having behavior issues)

If you update the underlying featureclass/shapefile, it should reflect those changes in the project's map when its opened and loaded.

[cmwrenn3]

Thanks, I'll see what I can do - I'm not sure how to access a map service via arcpy, but I'll try to find out.

[Anonymous User]

Knowing that you are doing this on a hosted layer now, I would look at using the arcgis for python api as a possible alternative.  The fact it worked in 3.6, this might be overkill if it is something that could get fixed in the next patch.  But, for example as an alternative: (This could very well not work right out of the box- I just pulled what I think would be the process from the editing features part of the API docs)

import arcpy
from arcgis.gis import GIS
import smtplib
from email.message import EmailMessage

gis = GIS(username="someuser", password="secret1234")

# Update COVID19 Zip Code data
def email_results(results):
    """Email the completion results."""
    msg = EmailMessage()
    msg.set_content(results)
    msg['Subject'] = 'COVID19 ZipCode Update Completed'
    msg['From'] = 'cmwrenn@co.pg.md.us'
    msg['To'] = 'cmwrenn@co.pg.md.us'
    server = smtplib.SMTP('smtp.co.pg.md.us', 25)
    server.starttls()
    server.send_message(msg)
    server.quit()

def getLayersFromAGOL():
    search_result = gis.content.search('title:COVID_STATEZIP') #?
    covidzip_item = search_result[0]
    covidCnt_layer = covidzip_item.layers

    covidCnt_fset = covidCnt_layer[0].query()
    covidZip_features = covidCnt_fset.features

    return covidZip_features

def getCounts():
    search_res = gis.content.search('title:ZIPCodes_MD') #?
    covidTot_item = search_res[0]
    covidTot_layer = covidTot_item.layers

    covidTot_fset = covidTot_layer[0].query()
    covidTotCnt_features = covidTot_fset.features

    return [f for f in covidTotCnt_features if f.attributes['ProtectedCount'][0]]

def updateZipcodeCounts(protectCnts, zipCodeCntLyr):

    return arcpy.CalculateFields_management(zipCodeCntLyr, "PYTHON3", [
        ["Cases", protectCnts],
        ['RATE', '!CASES! / !POP100! * 100000'],
        ['IMPRT_DATE', 'time.strftime("%m/%d/%Y")']])

if __name__ == '__main__':
    covidCntsLyr = getLayersFromAGOL()
    updtedLyr = updateZipcodeCounts(getCounts(), covidCntsLyr)

    covidCntsLyr.edit_features(updates=[updtedLyr])
    email_results('The ZipCodes have been updated with the current Cases and Population per 100,000 cases.')
    print('Completed') ## Flow Trace

[cmwrenn3]

I think, for now, we can run the script (via a DOS/Windows batch file) from a machine that has 2.6.3.  I'll contact Esri service in the mean time.

I will try the AGOL API angle, but with the interactive authentication (Auth2?); I'm not comfortable placing passwords as plain text inside a script.

Thanks.

[Anonymous User]

That is understandable and going the API route with hiding the sensitive info will certainly add some code as a solution.  Hopefully tech support can help.

To get around storing the username/password as text in scripts, one method we use is keeping them in a separate file that is located elsewhere on the server and is retrieved using a script that encrypts and decrypts the values.  Not 100% secure procedure, but it keeps the sensitive info from being in plain view within the script.  I can send you the encrypt/decrypt script if interested.