topic Re: 'ascii' codec can't encode character u'\u201c' in Python Questions

'ascii' codec can't encode character u'\u201c'

JoeBorgione — Fri, 10 Dec 2021 20:59:40 GMT

Database qa/qc would be so much easier when no data is entered....

Last week I was dealing with newline characters. (see Where clause for '\n' ). Today I'm getting the following error:

Runtime error 
Traceback (most recent call last):
  File "<string>", line 26, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 10: ordinal not in range(128)
‍‍‍‍

Okay... Best I can tell u201c is a left double quote.(http://www.fileformat.info/info/unicode/char/201C/index.htm ). I put

# -*- coding: utf-8 -*-   ‍

as the first line of my script and it still errors out. Am I doomed, or is there a way past these special characters?

Re: 'ascii' codec can't encode character u'\u201c'

DanPatterson_Retired — Tue, 12 Feb 2019 22:40:27 GMT

Joe,

what is the string? (post it if you can at least the first 12 or so characters

How are you 'getting it/reading it'

Sadly, if these things are embedded, sometimes they are hard to trap before you can encode/decode them

Re: 'ascii' codec can't encode character u'\u201c'

JoeBorgione — Tue, 12 Feb 2019 22:47:03 GMT

It's embedded in a field value. I don't 'see' it at all; that's the problem. From last week's exercise, I'm plowing through a list of tables, and for each text field I replace the \n with 'real' text character. My loop and update cursor work splendidly, until it gets to one field in the last table of the list....

Re: 'ascii' codec can't encode character u'\u201c'

JoshuaBixby — Tue, 12 Feb 2019 22:57:09 GMT

ArcPy Cursors return Unicode, even in ArcMap, so it isn't the data or the cursor. What code is generating this error message?

Re: 'ascii' codec can't encode character u'\u201c'

DanPatterson_Retired — Tue, 12 Feb 2019 23:10:31 GMT

Joe What is the origin of the tables you are working with? not dbase? are the gdb tables?

Re: 'ascii' codec can't encode character u'\u201c'

JoeBorgione — Fri, 10 Dec 2021 20:59:43 GMT

Joshua Bixby‌: I can run a search cursor and simply return the values to the screen and it works. If I send the output of the search cursor to a local text file it bombs. If I try an update cursor it bombs. Code below for each.

Dan Patterson‌: Initially the tables come from Survey 123 and and then are in an Enterprize GDB (SDE, SQL Server 2012 back end) I copy the data tables to a local File gdb for testing purposes.

As mentioned, I can loop through a list of tables and it's smooth sailing until I get to this one table. I picked the one field because when I run this in the Spyder console, I get this error:

  File "<ipython-input-2-d7b63bdc501d>", line 18, in <module>
    for updateRow in updateRows:

RuntimeError: The row contains a bad value. [SITENOTES]‍‍‍‍

The error I mentioned in my initial post came from an ArcMap 10.6.1 python window. Thanks guys.

# this does not fail:

import arcpy
arcpy.env.workspace = r'J:\WaterQuality\test_tables.gdb'
fields = ['OBJECTID', 'SITENOTES']
table = 'MacrosSamples'

with arcpy.da.SearchCursor(table,fields)as cursor:
    for row in cursor:
        if row[1] == None:
            pass
        elif "\n" in row[1]:
            print(row)
###########################################
# this fails Search cursor to text file for output in Spyder console or ArcMap Python Window:
# -*- coding: utf-8 -*-        
import arcpy,sys
#arcpy.env.workspace = r'I:\GIS\ArcSDE\SuperUser\pwengfc\SLCOen@pweng.sde' #r'J:\WaterQuality\test_tables.gdb'
arcpy.env.workspace = r'J:\WaterQuality\test_tables.gdb'
#tableList = ['BacteriaSamples', 'BacteriaBatch', 'BacteriaSamples', 'FieldParameters', 'MacrosSamples']
tableList = ['MacrosSamples']

logFile = r'C:\scripts\logs\newlines3.txt'
logoutput = open(logFile,'w')
sys.stdout = logoutput


for table in tableList:
    fieldNames = []
    tableFields = arcpy.ListFields(table,'','String')
    for f in tableFields:
        fieldNames.append(f.name)      
    for fieldName in fieldNames:
        #print('{} = table; {} = field name'.format(table,fieldName))
        field = [fieldName]
        with arcpy.da.SearchCursor(table,field)as cursor:
            for row in cursor:
                if row[0] == None:
                    pass
                elif "\n" in row[0]:
                    print('{} = tablename; {} = fieldvalue'.format(table,row[0]))
######################
# and this fails  Update Cursor  in Spyder console or ArcMap Python Window::
# -*- coding: utf-8 -*- 
import arcpy
#tableList = ['BacteriaBatch', 'BacteriaSamples', 'FieldParameters', 'MacrosSamples']
arcpy.env.workspace = r'J:\WaterQuality\test_tables.gdb'
ws = arcpy.env.workspace

edit = arcpy.da.Editor(ws)
edit.startEditing(False,False) #use False False: with undo,(False) multiuser mode(False)
edit.startOperation()
tableList = ['MacrosSamples']
for table in tableList:
    fieldNames = []
    tableFields = arcpy.ListFields(table,'','String')
    for f in tableFields:
        fieldNames.append(f.name)
        for fieldName in fieldNames:
            field = [fieldName]
        with arcpy.da.UpdateCursor(table,field) as updateRows:
            for updateRow in updateRows:
                if updateRow[0]== None:
                    pass
                elif "\n" in updateRow[0]:
                    updateRow[0] = updateRow[0].replace("\n","=>")
                    updateRows.updateRow(updateRow)
                    print('Updated {}  {}'.format(table,field))    
edit.stopOperation()
edit.stopEditing(True)
########################‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: 'ascii' codec can't encode character u'\u201c'

DanPatterson_Retired — Wed, 13 Feb 2019 00:02:40 GMT

in 10.6.1 … aren't you using python 2.7 then??

which isn't python 3.6,

which means it isn't Unicode aware without 'efforts"

Re: 'ascii' codec can't encode character u'\u201c'

JoeBorgione — Wed, 13 Feb 2019 00:39:26 GMT

But from the spyder console it's running 3.x, no?

Re: 'ascii' codec can't encode character u'\u201c'

RandyBurton — Fri, 10 Dec 2021 20:59:46 GMT

On occasion, I've used a conversion dictionary as a "hammer":

conversion = {
    u"\u2019": "'", # apostrophe
    u"\u201c": "\"", # double quote
    u"\u00a0": " ", # non breaking space
    "\n" : "=>" # newline
    }

def convert(data):
    for k, v in conversion.items():
        # https://stackoverflow.com/questions/14156473
        data = data.replace(k,v)
    return data.encode('ascii')

print convert("hello\nworld" )
# hello=>world

print convert(u"hello \u201cworld\u201c" )
# hello "world"‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: 'ascii' codec can't encode character u'\u201c'

Luke_Pinner — Wed, 13 Feb 2019 01:38:28 GMT

Joe Borgione wrote:
But from the spyder console it's running 3.x, no?

No. Python 2.7.

Though if you actually are using Python 3.6 and arcpy in spyder, you're using ArcGIS Pro arcpy, not ArcGIS Desktop arcpy.

Re: 'ascii' codec can't encode character u'\u201c'

Luke_Pinner — Fri, 10 Dec 2021 20:59:49 GMT

# -*- coding: utf-8 -*-

This only tells the python interpreter that string literals in your script are utf-8, it doesn't apply to string data that you use in the script.

i.e

status = "It's -10°C here in Überwald and I'm sitting outside at the Café"
print status‍‍

C:\Python27\ArcGIS10.3\python.exe test.py
  File "test.py", line 1
SyntaxError: Non-ASCII character '\xc2' in file test.py on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details‍‍‍

# -*- coding: utf-8 -*-

status = "It's -10°C here in Überwald and I'm sitting outside at the Café"
print status‍‍‍‍

C:\Python27\ArcGIS10.3\python.exe test.py
It's -10°C here in Überwald and I'm sitting outside at the Café

Process finished with exit code 0
‍‍‍‍

Re: 'ascii' codec can't encode character u'\u201c'

JoeBorgione — Wed, 13 Feb 2019 13:11:11 GMT

I use Spyder as my IDE; it was added as a package to my cloned ArcGIS Pro 2.3 env. However, if/when I get issues like I'm having, I'll copy and paste my code into an ArcGIS Arcmap or ArcCatalog python window just to see what happens there. Am I incorrect that the pythonw.exe I point to when executing Spyder is 3.x? see: _ModuleMock object has to attribute 'PLUGIN CLASS' I expect theArcMap/ArcCatalog python window to use a 2.x interpreter...

edited now that I'm at work: here is my clone path and pythonw.exe info:

Re: 'ascii' codec can't encode character u'\u201c'

JoeBorgione — Wed, 13 Feb 2019 13:19:27 GMT

Hammers... Fixing the world's problems one smack at a time! 😉

Re: 'ascii' codec can't encode character u'\u201c'

JoeBorgione — Fri, 10 Dec 2021 20:59:52 GMT

I took Randy's hammer idea and got the following. I can actually get a handle as to which records are the offenders:

""" Run in an ArcMap 10.6.1 Python window.  I got the same output in
    a Spyder console
"""
>>> import arcpy
arcpy.env.workspace = r'J:\WaterQuality\test_tables.gdb'
#arcpy.env.workspace = r'I:\GIS\ArcSDE\SuperUser\pwengfc\SLCOen@pweng.sde'
fields = ['OBJECTID', 'SITENOTES']
table = 'MacrosSamples'

with arcpy.da.SearchCursor(table,fields)as cursor:
    for row in cursor:
        if row[1] == None:
            pass
        elif u"\u201c" in row[1]:
            print(row)
(3, u'Substrate \u201co\u201d = bedrock\nUpper reaches definitely take caution, high flow at this time. Lower reaches nice and easy to sample. Slick rocks. Better with multiple samplers.')
(16, u'Park at Spruces campground lot.\nX site (A) is just downstream of bridge. Work downstream from \u201cA\u201d site.\nSome rocks are very slippery\nSeveral large logs/debris jams in reach')
(33, u'\u201cBedrock\u201d is a calcified structure.\nSafe for 1-2 samplers\nGolfers utilizing course that river runs through\nCulvert at start and middle of reach\nEasy parking in golf course lot')
(65, u'Substrate \u201cother\u201d = bedrock')
(66, u'Substrate \u201cother\u201d = bedrock')
(69, u'Substrate \u201co\u201d = bedrock\nSafe for 1 sampler')
(81, u'Lambs Canyon Creek enters Parleys at beginning of \u201cA\u201d transect. Long hike to x site.  Safe for 1 person.\n')
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: 'ascii' codec can't encode character u'\u201c'

JoshuaBixby — Wed, 13 Feb 2019 16:27:01 GMT

If you are working in Python 3, try specifying the encoding when you open the file. From Open - Built-in Functions — Python 3.7.2 documentation

encoding is the name of the encoding used to decode or encode the file. This should only be used in text mode. The default encoding is platform dependent (whatever locale.getpreferredencoding() returns), but any text encoding supported by Python can be used. See the codecs module for the list of supported encodings.

In Python 2, you can use the 15.2. io — Core tools for working with streams — Python 2.7.15 documentation module that allows specifying the text file encoding.

Re: 'ascii' codec can't encode character u'\u201c'

Luke_Pinner — Thu, 14 Feb 2019 04:22:19 GMT

Mixing and matching is going to cause you grief and confuse us . I said it was Python 2.7 as I'd read you were running your code in in 10.6.1.

You can install Spyder for Desktop (python 2.7) and Spyder for Pro (python 3)

Re: 'ascii' codec can't encode character u'\u201c'

JoeBorgione — Thu, 14 Feb 2019 13:13:08 GMT

Agreed. It can be confusing and often times is. We have a rather large collection of 2.7 scripts that we are in the process of migrating, finding gotchas all along the way.

This particular exercise may be one of futility as it seems everytime I check for one embedded special character, I error out on another regardless of python version...

Re: 'ascii' codec can't encode character u'\u201c'

DanPatterson_Retired — Thu, 14 Feb 2019 13:48:29 GMT

Some distractions for you Joe

http://ptgmedia.pearsoncmg.com/imprint_downloads/informit/promotions/python/python2python3.pdf

Re: 'ascii' codec can't encode character u'\u201c'

JoeBorgione — Thu, 14 Feb 2019 14:50:30 GMT

Nice. Thanks. I can use a distraction...

Re: 'ascii' codec can't encode character u'\u201c'

Luke_Pinner — Fri, 10 Dec 2021 20:59:54 GMT

Can you explain a bit more about what you want to do with this text data. Your errors are happening when you print it out and this is expected (in Python 2, though in Python 3 you don't need to worry as much as strings are unicode objects).

For example: I can read in some data with non-ascii characters and write it out to another table with no issues, but if I try to print it (or write it to a text file/spreadsheet) without encoding it, I get the dreaded UnicodeEncodeError. If you want to print it/output it you need to encode it.

table,field = 'c:/temp/default.gdb/test', 'testfield'

with arcpy.da.SearchCursor(table,field) as rows:
    for row in rows:
        data = row[0]
        break
        
print('{} = tablename; {} = fieldvalue'.format(table,data))

# Runtime error 
# Traceback (most recent call last):
#   File "<string>", line 4, in <module>
# UnicodeEncodeError: 'ascii' codec can't encode character u'\xb0' in position 8: ordinal not in range(128)

with open('c:/temp/test.txt', 'w') as t:
    t.write(data)

# Runtime error
# Traceback (most recent call last):
#   File "<string>", line 2, in <module>
# UnicodeEncodeError: 'ascii' codec can't encode character u'\xb0' in position 8: ordinal not in range(128)

print('{} = tablename; {} = fieldvalue'.format(table,data.encode('utf-8')))
# c:/temp/default.gdb/test = tablename; It's -10°C here in Überwald and I'm sitting outside at the Café = fieldvalue
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Hammers sometimes miss and hit you on the thumb. You can try and replace most of the usual non-ASCII characters, but you'll always run across another...

Your best bet is deal with it as unicode then encode it on output.

If you absolutely must force it to ascii, t ry the hammer (manual replacement), if that fails, just strip them out:

data.encode('ascii', 'ignore')

You could also look at a 3rd party library that takes Unicode data and tries to represent it in ASCII characters - Unidecode · PyPI