'ascii' codec can't encode character u'\u201c'

[JoeBorgione]

Database qa/qc would be so much easier when no data is entered....

Last week I was dealing with newline characters. (see Where clause for '\n' ).  Today I'm getting the following error:

Runtime error 
Traceback (most recent call last):
  File "<string>", line 26, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 10: ordinal not in range(128)
‍‍‍‍

Okay...  Best I can tell u201c is a left double quote.(http://www.fileformat.info/info/unicode/char/201C/index.htm ). I put

# -*- coding: utf-8 -*-   ‍

as the first line of my script and it still errors out.  Am I doomed, or is there a way past these special characters?

[DanPatterson_Retired]

Joe,

what is the string?  (post it if you can at least the first 12 or so characters

How are you 'getting it/reading it'

Sadly, if these things are embedded, sometimes they are hard to trap before you can encode/decode them

[JoeBorgione]

It's embedded in a field value.  I don't 'see' it at all; that's the problem.  From last week's exercise, I'm plowing through a list of tables, and for each text field I replace the \n with  'real' text character.  My loop and update cursor work splendidly, until it gets to one field in the last table of the list....

[JoshuaBixby]

ArcPy Cursors return Unicode, even in ArcMap, so it isn't the data or the cursor.  What code is generating this error message?

[DanPatterson_Retired]

Joe What is the origin of the tables you are working with? not dbase? are the gdb tables?

[JoeBorgione]

Joshua Bixby‌: I can run a search cursor and simply return the values to the screen and it works.  If I send the output of the search cursor to a local text file it bombs.  If I try an update cursor it bombs.  Code below for each.

Dan Patterson‌: Initially the tables come from Survey 123 and and then are in an Enterprize GDB (SDE, SQL Server 2012 back end)  I copy the data tables to a local File gdb for testing purposes.

As mentioned, I can loop through a list of tables and it's smooth sailing until I get to this one table.  I picked the one field because when I run this in the Spyder console, I get this error:

  File "<ipython-input-2-d7b63bdc501d>", line 18, in <module>
    for updateRow in updateRows:

RuntimeError: The row contains a bad value. [SITENOTES]‍‍‍‍

The error I mentioned in my initial post came from an ArcMap 10.6.1 python window.  Thanks guys.

# this does not fail:

import arcpy
arcpy.env.workspace = r'J:\WaterQuality\test_tables.gdb'
fields = ['OBJECTID', 'SITENOTES']
table = 'MacrosSamples'

with arcpy.da.SearchCursor(table,fields)as cursor:
    for row in cursor:
        if row[1] == None:
            pass
        elif "\n" in row[1]:
            print(row)
###########################################
# this fails Search cursor to text file for output in Spyder console or ArcMap Python Window:
# -*- coding: utf-8 -*-        
import arcpy,sys
#arcpy.env.workspace = r'I:\GIS\ArcSDE\SuperUser\pwengfc\SLCOen@pweng.sde' #r'J:\WaterQuality\test_tables.gdb'
arcpy.env.workspace = r'J:\WaterQuality\test_tables.gdb'
#tableList = ['BacteriaSamples', 'BacteriaBatch', 'BacteriaSamples', 'FieldParameters', 'MacrosSamples']
tableList = ['MacrosSamples']

logFile = r'C:\scripts\logs\newlines3.txt'
logoutput = open(logFile,'w')
sys.stdout = logoutput


for table in tableList:
    fieldNames = []
    tableFields = arcpy.ListFields(table,'','String')
    for f in tableFields:
        fieldNames.append(f.name)      
    for fieldName in fieldNames:
        #print('{} = table; {} = field name'.format(table,fieldName))
        field = [fieldName]
        with arcpy.da.SearchCursor(table,field)as cursor:
            for row in cursor:
                if row[0] == None:
                    pass
                elif "\n" in row[0]:
                    print('{} = tablename; {} = fieldvalue'.format(table,row[0]))
######################
# and this fails  Update Cursor  in Spyder console or ArcMap Python Window::
# -*- coding: utf-8 -*- 
import arcpy
#tableList = ['BacteriaBatch', 'BacteriaSamples', 'FieldParameters', 'MacrosSamples']
arcpy.env.workspace = r'J:\WaterQuality\test_tables.gdb'
ws = arcpy.env.workspace

edit = arcpy.da.Editor(ws)
edit.startEditing(False,False) #use False False: with undo,(False) multiuser mode(False)
edit.startOperation()
tableList = ['MacrosSamples']
for table in tableList:
    fieldNames = []
    tableFields = arcpy.ListFields(table,'','String')
    for f in tableFields:
        fieldNames.append(f.name)
        for fieldName in fieldNames:
            field = [fieldName]
        with arcpy.da.UpdateCursor(table,field) as updateRows:
            for updateRow in updateRows:
                if updateRow[0]== None:
                    pass
                elif "\n" in updateRow[0]:
                    updateRow[0] = updateRow[0].replace("\n","=>")
                    updateRows.updateRow(updateRow)
                    print('Updated {}  {}'.format(table,field))    
edit.stopOperation()
edit.stopEditing(True)
########################‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[DanPatterson_Retired]

in 10.6.1 … aren't you using python 2.7 then??

which isn't python 3.6,

which means it isn't Unicode aware without 'efforts"

[JoeBorgione]

But from the spyder console it's running 3.x, no?

[Luke_Pinner]

Joe Borgione wrote:

But from the spyder console it's running 3.x, no?

No. Python 2.7. 

Though if you actually are using Python 3.6 and arcpy in spyder, you're using ArcGIS Pro arcpy, not ArcGIS Desktop arcpy.

[JoeBorgione]

I use Spyder as my IDE; it was added as a package to my cloned ArcGIS Pro 2.3 env.  However, if/when I get issues like I'm having, I'll copy and paste my code into an ArcGIS Arcmap or ArcCatalog python window just to see what happens there.  Am I incorrect that the pythonw.exe I point to when executing Spyder is 3.x?  see: _ModuleMock object has to attribute 'PLUGIN CLASS'    I expect  theArcMap/ArcCatalog python window to use a 2.x interpreter...

edited now that I'm at work:  here is my clone path and pythonw.exe info: