Database qa/qc would be so much easier when no data is entered.... Last week I was dealing with newline characters. (see Where clause for '\n' ). Today I'm getting the following error: Runtime error Traceback ( most recent call last ) : File " " , line 26 , in module > UnicodeEncodeError : 'ascii' codec can 't encode character u' \u201c' in position 10 : ordinal not in range ( 128 ) Okay... Best I can tell u201c is a left double quote.(http://www.fileformat.info/info/unicode/char/201C/index.htm ). I put # -*- coding: utf-8 -*- as the first line of my script and it still errors out. Am I doomed, or is there a way past these special characters?

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

'ascii' codec can't encode character u'\u201c'

JoeBorgione

Database qa/qc would be so much easier when no data is entered....

Last week I was dealing with newline characters. (see Where clause for '\n' ). Today I'm getting the following error:

Runtime error 
Traceback <SPAN class="punctuation token">(</SPAN>most recent call last<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
  File <SPAN class="string token">"<string>"</SPAN><SPAN class="punctuation token">,</SPAN> line <SPAN class="number token">26</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="keyword token">in</SPAN> <SPAN class="operator token"><</SPAN>module<SPAN class="operator token">></SPAN>
UnicodeEncodeError<SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'ascii'</SPAN> codec can<SPAN class="string token">'t encode character u'</SPAN>\u201c' <SPAN class="keyword token">in</SPAN> position <SPAN class="number token">10</SPAN><SPAN class="punctuation token">:</SPAN> ordinal <SPAN class="operator token">not</SPAN> <SPAN class="keyword token">in</SPAN> range<SPAN class="punctuation token">(</SPAN><SPAN class="number token">128</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Okay... Best I can tell u201c is a left double quote.(http://www.fileformat.info/info/unicode/char/201C/index.htm ). I put

<SPAN class="comment token"># -*- coding: utf-8 -*-   </SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN></SPAN>

as the first line of my script and it still errors out. Am I doomed, or is there a way past these special characters?

Najděte více příspěvků označených jako

Accepted answers

All comments

Luke_Pinner

Joe Borgione wrote:
# -*- coding: utf-8 -*-   """ This has always been my very first line """" I would have thought that line 1 would have taken care of things, but I guess not..

So what's the answer? I guess it depends.....

As I mentioned earlier the special coding: comment line only applies to string literals in your code, i.e literally written in the .py file, not to any data/variables.

The real answer is to treat your strings as unicode (cos they are when read from a cursor) and do any string munging with unicode strings, i.e univar = row[0].replace(u'\n', u'==>') then encode on output univar.encode('utf8').

JoeBorgione

Yeah but..... Our customer base is pretty broad, lots of ArcMap ArcCatalog users, and a lot of them have theirs heels dug in pretty deep. So as sys admin types, myself and the others my team have to play to both sides of the fence. We run tons of overnight scheduled tasks for individual agency eGDBs and this will be one of them, now that I know how to avoid the errors I was encountering. It's a coin toss which way to go.

The real issue though is this interim time where python 2.x and 3.x are overlapping along with respective Arcpy functionalities. Job security through the remainder of my career I suppose...

DanPatterson_Retired

quit using arcmap I suppose

JoeBorgione

Finally got resolution to this on, thanks to Joshua at ESRI tech support. He had me add a couple of lines to my script and all is well...

<SPAN class="comment token"># -*- coding: utf-8 -*-   """ This has always been my very first line """"</SPAN>

<SPAN class="keyword token">import</SPAN> arcpy<SPAN class="punctuation token">,</SPAN>time<SPAN class="punctuation token">,</SPAN>datetime<SPAN class="punctuation token">,</SPAN>os<SPAN class="punctuation token">,</SPAN>sys
<SPAN class="string token">""" added the two lines below """</SPAN>
reload<SPAN class="punctuation token">(</SPAN>sys<SPAN class="punctuation token">)</SPAN>             
sys<SPAN class="punctuation token">.</SPAN>setdefaultencoding<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'utf-8'</SPAN><SPAN class="punctuation token">)</SPAN>‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

I would have thought that line 1 would have taken care of things, but I guess not..

There is a gotcha though: I've added lines 5 and 6 to my script that is the basis of a script tool used in ArcMap or Catalog 10.6.1. If I use those two lines in a standalone script and execute from a Spyder console (associated with python 3.x) it fails. See this stackoverflow post that states:

Also, the use of sys.setdefaultencoding() has always been discouraged, and it has become a no-op in py3k. The encoding of py3k is hard-wired to "utf-8" and changing it raises an error.

So what's the answer? I guess it depends.....

DanPatterson_Retired

python 3.6.8

uni <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN>u<SPAN class="string token">"\u2019"</SPAN><SPAN class="punctuation token">,</SPAN>  u<SPAN class="string token">"\u201c"</SPAN><SPAN class="punctuation token">,</SPAN> u<SPAN class="string token">"\u00a0"</SPAN><SPAN class="punctuation token">,</SPAN> u<SPAN class="string token">"Überwald"</SPAN><SPAN class="punctuation token">,</SPAN>
 u'Substrate \u201co\u201d <SPAN class="operator token">=</SPAN> bedrock\nUpper reaches definitely take caution<SPAN class="punctuation token">,</SPAN>
 high flow at this time<SPAN class="punctuation token">.</SPAN> Lower reaches nice <SPAN class="operator token">and</SPAN> easy to sample<SPAN class="punctuation token">.</SPAN> Slick rocks<SPAN class="punctuation token">.</SPAN> Better <SPAN class="keyword token">with</SPAN> multiple samplers<SPAN class="punctuation token">.</SPAN>'<SPAN class="punctuation token">,</SPAN>
 u<SPAN class="string token">'Park at Spruces campground lot.\nX site (A) is just downstream of bridge. Work downstream from \u201cA\u201d site.\nSome rocks are very slippery\nSeveral large logs/debris jams in reach'</SPAN><SPAN class="punctuation token">,</SPAN>
 u<SPAN class="string token">'\u201cBedrock\u201d is a calcified structure.\nSafe for 1-2 samplers\nGolfers utilizing course that river runs through\nCulvert at start and middle of reach\nEasy parking in golf course lot'</SPAN><SPAN class="punctuation token">,</SPAN>
 u<SPAN class="string token">'Substrate \u201cother\u201d = bedrock'</SPAN><SPAN class="punctuation token">,</SPAN>
 u<SPAN class="string token">'Substrate \u201cother\u201d = bedrock'</SPAN><SPAN class="punctuation token">,</SPAN>
 u<SPAN class="string token">'Substrate \u201co\u201d = bedrock\nSafe for 1 sampler'</SPAN><SPAN class="punctuation token">,</SPAN>
 u<SPAN class="string token">'Lambs Canyon Creek enters Parleys at beginning of \u201cA\u201d transect. Long hike to x site.  Safe for 1 person.\n'</SPAN><SPAN class="punctuation token">]</SPAN>

<SPAN class="keyword token">for</SPAN> i <SPAN class="keyword token">in</SPAN> uni<SPAN class="punctuation token">:</SPAN>
    <SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN>i<SPAN class="punctuation token">.</SPAN>encode<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>decode<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
    
<SPAN class="operator token">-</SPAN>  ’
<SPAN class="operator token">-</SPAN>  “
<SPAN class="operator token">-</SPAN>   
<SPAN class="operator token">-</SPAN>  Überwald
<SPAN class="operator token">-</SPAN>  Substrate “o” <SPAN class="operator token">=</SPAN> bedrock
Upper reaches definitely take caution<SPAN class="punctuation token">,</SPAN> high flow at this time<SPAN class="punctuation token">.</SPAN> Lower reaches nice <SPAN class="operator token">and</SPAN> easy to sample<SPAN class="punctuation token">.</SPAN> Slick rocks<SPAN class="punctuation token">.</SPAN> Better <SPAN class="keyword token">with</SPAN> multiple samplers<SPAN class="punctuation token">.</SPAN>
<SPAN class="operator token">-</SPAN>  Park at Spruces campground lot<SPAN class="punctuation token">.</SPAN>
X site <SPAN class="punctuation token">(</SPAN>A<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">is</SPAN> just downstream of bridge<SPAN class="punctuation token">.</SPAN> Work downstream <SPAN class="keyword token">from</SPAN> “A” site<SPAN class="punctuation token">.</SPAN>
Some rocks are very slippery
Several large logs<SPAN class="operator token">/</SPAN>debris jams <SPAN class="keyword token">in</SPAN> reach
<SPAN class="operator token">-</SPAN>  “Bedrock” <SPAN class="keyword token">is</SPAN> a calcified structure<SPAN class="punctuation token">.</SPAN>
Safe <SPAN class="keyword token">for</SPAN> <SPAN class="number token">1</SPAN><SPAN class="operator token">-</SPAN><SPAN class="number token">2</SPAN> samplers
Golfers utilizing course that river runs through
Culvert at start <SPAN class="operator token">and</SPAN> middle of reach
Easy parking <SPAN class="keyword token">in</SPAN> golf course lot
<SPAN class="operator token">-</SPAN>  Substrate “other” <SPAN class="operator token">=</SPAN> bedrock
<SPAN class="operator token">-</SPAN>  Substrate “other” <SPAN class="operator token">=</SPAN> bedrock
<SPAN class="operator token">-</SPAN>  Substrate “o” <SPAN class="operator token">=</SPAN> bedrock
Safe <SPAN class="keyword token">for</SPAN> <SPAN class="number token">1</SPAN> sampler
<SPAN class="operator token">-</SPAN>  Lambs Canyon Creek enters Parleys at beginning of “A” transect<SPAN class="punctuation token">.</SPAN> Long hike to x site<SPAN class="punctuation token">.</SPAN>  Safe <SPAN class="keyword token">for</SPAN> <SPAN class="number token">1</SPAN> person<SPAN class="punctuation token">.</SPAN>

<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

DanPatterson_Retired

I suspect \ in the field, turns a lot of stuff into unicode

JoeBorgione

You can try and replace most of the usual non-ASCII characters, but you'll always run across another... And that's exactly what I encountered. The field I need to edit is a 'comments/field notes' field so just about anything and everything is 'allowed'.

Ultimately, I just need to replace the newline with another symbol in the database field. I'm not at work today so I don't have the code in front of me but I tried either the decode or encode method with an ignore argument. It didn't quite work out; I must still have a flaw in my business logic. Thankfully this project got pushed to a back burner in favor of some others.

I'll give the encode(ascii, ignore) method approach when I get back to work. Thanks!

Luke_Pinner

Can you explain a bit more about what you want to do with this text data. Your errors are happening when you print it out and this is expected (in Python 2, though in Python 3 you don't need to worry as much as strings are unicode objects).

For example: I can read in some data with non-ascii characters and write it out to another table with no issues, but if I try to print it (or write it to a text file/spreadsheet) without encoding it, I get the dreaded UnicodeEncodeError. If you want to print it/output it you need to encode it.

table<SPAN class="punctuation token">,</SPAN>field <SPAN class="operator token">=</SPAN> <SPAN class="string token">'c:/temp/default.gdb/test'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'testfield'</SPAN>

<SPAN class="keyword token">with</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>table<SPAN class="punctuation token">,</SPAN>field<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> rows<SPAN class="punctuation token">:</SPAN>
    <SPAN class="keyword token">for</SPAN> row <SPAN class="keyword token">in</SPAN> rows<SPAN class="punctuation token">:</SPAN>
        data <SPAN class="operator token">=</SPAN> row<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN>
        <SPAN class="keyword token">break</SPAN>
        
<SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">'{} = tablename; {} = fieldvalue'</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN>table<SPAN class="punctuation token">,</SPAN>data<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>

<SPAN class="comment token"># Runtime error </SPAN>
<SPAN class="comment token"># Traceback (most recent call last):</SPAN>
<SPAN class="comment token">#   File "<string>", line 4, in <module></SPAN>
<SPAN class="comment token"># UnicodeEncodeError: 'ascii' codec can't encode character u'\xb0' in position 8: ordinal not in range(128)</SPAN>

<SPAN class="keyword token">with</SPAN> open<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'c:/temp/test.txt'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'w'</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> t<SPAN class="punctuation token">:</SPAN>
    t<SPAN class="punctuation token">.</SPAN>write<SPAN class="punctuation token">(</SPAN>data<SPAN class="punctuation token">)</SPAN>

<SPAN class="comment token"># Runtime error</SPAN>
<SPAN class="comment token"># Traceback (most recent call last):</SPAN>
<SPAN class="comment token">#   File "<string>", line 2, in <module></SPAN>
<SPAN class="comment token"># UnicodeEncodeError: 'ascii' codec can't encode character u'\xb0' in position 8: ordinal not in range(128)</SPAN>

<SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">'{} = tablename; {} = fieldvalue'</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN>table<SPAN class="punctuation token">,</SPAN>data<SPAN class="punctuation token">.</SPAN>encode<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'utf-8'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token"># c:/temp/default.gdb/test = tablename; It's -10°C here in Überwald and I'm sitting outside at the Café = fieldvalue</SPAN>
<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Hammers sometimes miss and hit you on the thumb. You can try and replace most of the usual non-ASCII characters, but you'll always run across another...

Your best bet is deal with it as unicode then encode it on output.

If you absolutely must force it to ascii, t ry the hammer (manual replacement), if that fails, just strip them out:

<SPAN class="" style="color: #999999; border: 0px; font-weight: inherit; font-size: 14px;">data.</SPAN>encode<SPAN class="" style="color: #999999; border: 0px; font-weight: inherit; font-size: 14px;">(</SPAN><SPAN class="" style="color: #669900; border: 0px; font-weight: inherit; font-size: 14px;">'ascii'</SPAN><SPAN class="" style="color: #999999; border: 0px; font-weight: inherit; font-size: 14px;">,</SPAN> <SPAN class="" style="color: #669900; border: 0px; font-weight: inherit; font-size: 14px;">'ignore'</SPAN><SPAN class="" style="color: #999999; border: 0px; font-weight: inherit; font-size: 14px;">)</SPAN>

You could also look at a 3rd party library that takes Unicode data and tries to represent it in ASCII characters - Unidecode · PyPI

bixb0012

If you are working in Python 3, try specifying the encoding when you open the file. From Open - Built-in Functions — Python 3.7.2 documentation

encoding is the name of the encoding used to decode or encode the file. This should only be used in text mode. The default encoding is platform dependent (whatever locale.getpreferredencoding() returns), but any text encoding supported by Python can be used. See the codecs module for the list of supported encodings.

In Python 2, you can use the 15.2. io — Core tools for working with streams — Python 2.7.15 documentation module that allows specifying the text file encoding.

JoeBorgione

I took Randy's hammer idea and got the following. I can actually get a handle as to which records are the offenders:

<SPAN class="string token">""" Run in an ArcMap 10.6.1 Python window.  I got the same output in
    a Spyder console
"""</SPAN>
<SPAN class="operator token">>></SPAN><SPAN class="operator token">></SPAN> <SPAN class="keyword token">import</SPAN> arcpy
arcpy<SPAN class="punctuation token">.</SPAN>env<SPAN class="punctuation token">.</SPAN>workspace <SPAN class="operator token">=</SPAN> r<SPAN class="string token">'J:\WaterQuality\test_tables.gdb'</SPAN>
<SPAN class="comment token">#arcpy.env.workspace = r'I:\GIS\ArcSDE\SuperUser\pwengfc\SLCOen@pweng.sde'</SPAN>
fields <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="string token">'OBJECTID'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'SITENOTES'</SPAN><SPAN class="punctuation token">]</SPAN>
table <SPAN class="operator token">=</SPAN> <SPAN class="string token">'MacrosSamples'</SPAN>

<SPAN class="keyword token">with</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>table<SPAN class="punctuation token">,</SPAN>fields<SPAN class="punctuation token">)</SPAN><SPAN class="keyword token">as</SPAN> cursor<SPAN class="punctuation token">:</SPAN>
    <SPAN class="keyword token">for</SPAN> row <SPAN class="keyword token">in</SPAN> cursor<SPAN class="punctuation token">:</SPAN>
        <SPAN class="keyword token">if</SPAN> row<SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">==</SPAN> None<SPAN class="punctuation token">:</SPAN>
            <SPAN class="keyword token">pass</SPAN>
        <SPAN class="keyword token">elif</SPAN> u<SPAN class="string token">"\u201c"</SPAN> <SPAN class="keyword token">in</SPAN> row<SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">:</SPAN>
            <SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN>row<SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="number token">3</SPAN><SPAN class="punctuation token">,</SPAN> u<SPAN class="string token">'Substrate \u201co\u201d = bedrock\nUpper reaches definitely take caution, high flow at this time. Lower reaches nice and easy to sample. Slick rocks. Better with multiple samplers.'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="number token">16</SPAN><SPAN class="punctuation token">,</SPAN> u<SPAN class="string token">'Park at Spruces campground lot.\nX site (A) is just downstream of bridge. Work downstream from \u201cA\u201d site.\nSome rocks are very slippery\nSeveral large logs/debris jams in reach'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="number token">33</SPAN><SPAN class="punctuation token">,</SPAN> u<SPAN class="string token">'\u201cBedrock\u201d is a calcified structure.\nSafe for 1-2 samplers\nGolfers utilizing course that river runs through\nCulvert at start and middle of reach\nEasy parking in golf course lot'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="number token">65</SPAN><SPAN class="punctuation token">,</SPAN> u<SPAN class="string token">'Substrate \u201cother\u201d = bedrock'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="number token">66</SPAN><SPAN class="punctuation token">,</SPAN> u<SPAN class="string token">'Substrate \u201cother\u201d = bedrock'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="number token">69</SPAN><SPAN class="punctuation token">,</SPAN> u<SPAN class="string token">'Substrate \u201co\u201d = bedrock\nSafe for 1 sampler'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">(</SPAN><SPAN class="number token">81</SPAN><SPAN class="punctuation token">,</SPAN> u<SPAN class="string token">'Lambs Canyon Creek enters Parleys at beginning of \u201cA\u201d transect. Long hike to x site.  Safe for 1 person.\n'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

JoeBorgione

Hammers... Fixing the world's problems one smack at a time! 😉

Luke_Pinner

<SPAN class="" style="color: slategray; border: 0px; font-weight: inherit;"># -*- coding: utf-8 -*-&nbsp;&nbsp; </SPAN>

This only tells the python interpreter that string literals in your script are utf-8, it doesn't apply to string data that you use in the script.

i.e

status <SPAN class="operator token">=</SPAN> <SPAN class="string token">"It's -10°C here in Überwald and I'm sitting outside at the Café"</SPAN>
<SPAN class="keyword token">print</SPAN> status<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

C:\Python27\ArcGIS10.3\python.exe test.py
  File "test.py", line 1
SyntaxError: Non-ASCII character '\xc2' in file test.py on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

<SPAN class="comment token"># -*- coding: utf-8 -*-</SPAN>

status <SPAN class="operator token">=</SPAN> <SPAN class="string token">"It's -10°C here in Überwald and I'm sitting outside at the Café"</SPAN>
<SPAN class="keyword token">print</SPAN> status<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

C:\Python27\ArcGIS10.3\python.exe test.py
It's -10°C here in Überwald and I'm sitting outside at the Café

Process finished with exit code 0
<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Luke_Pinner

Joe Borgione wrote:
But from the spyder console it's running 3.x, no?

No. Python 2.7.

Though if you actually are using Python 3.6 and arcpy in spyder, you're using ArcGIS Pro arcpy, not ArcGIS Desktop arcpy.

RandyBurton

On occasion, I've used a conversion dictionary as a "hammer":

conversion = {
    u"\u2019": "'", # apostrophe
    u"\u201c": "\"", # double quote
    u"\u00a0": " ", # non breaking space
    "\n" : "=>" # newline
    }

def convert(data):
    for k, v in conversion.items():
        # https://stackoverflow.com/questions/14156473
        data = data.replace(k,v)
    return data.encode('ascii')

print convert("hello\nworld" )
# hello=>world

print convert(u"hello \u201cworld\u201c" )
# hello "world"<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

JoeBorgione

But from the spyder console it's running 3.x, no?

DanPatterson_Retired

in 10.6.1 … aren't you using python 2.7 then??

which isn't python 3.6,

which means it isn't Unicode aware without 'efforts"

JoeBorgione

Joshua Bixby‌: I can run a search cursor and simply return the values to the screen and it works. If I send the output of the search cursor to a local text file it bombs. If I try an update cursor it bombs. Code below for each.

Dan Patterson‌: Initially the tables come from Survey 123 and and then are in an Enterprize GDB (SDE, SQL Server 2012 back end) I copy the data tables to a local File gdb for testing purposes.

As mentioned, I can loop through a list of tables and it's smooth sailing until I get to this one table. I picked the one field because when I run this in the Spyder console, I get this error:

  File <SPAN class="string token">"<ipython-input-2-d7b63bdc501d>"</SPAN><SPAN class="punctuation token">,</SPAN> line <SPAN class="number token">18</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="keyword token">in</SPAN> <SPAN class="operator token"><</SPAN>module<SPAN class="operator token">></SPAN>
    <SPAN class="keyword token">for</SPAN> updateRow <SPAN class="keyword token">in</SPAN> updateRows<SPAN class="punctuation token">:</SPAN>

RuntimeError<SPAN class="punctuation token">:</SPAN> The row contains a bad value<SPAN class="punctuation token">.</SPAN> <SPAN class="punctuation token">[</SPAN>SITENOTES<SPAN class="punctuation token">]</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

The error I mentioned in my initial post came from an ArcMap 10.6.1 python window. Thanks guys.

<SPAN class="comment token"># this does not fail:</SPAN>

<SPAN class="keyword token">import</SPAN> arcpy
arcpy<SPAN class="punctuation token">.</SPAN>env<SPAN class="punctuation token">.</SPAN>workspace <SPAN class="operator token">=</SPAN> r<SPAN class="string token">'J:\WaterQuality\test_tables.gdb'</SPAN>
fields <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="string token">'OBJECTID'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'SITENOTES'</SPAN><SPAN class="punctuation token">]</SPAN>
table <SPAN class="operator token">=</SPAN> <SPAN class="string token">'MacrosSamples'</SPAN>

<SPAN class="keyword token">with</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>table<SPAN class="punctuation token">,</SPAN>fields<SPAN class="punctuation token">)</SPAN><SPAN class="keyword token">as</SPAN> cursor<SPAN class="punctuation token">:</SPAN>
    <SPAN class="keyword token">for</SPAN> row <SPAN class="keyword token">in</SPAN> cursor<SPAN class="punctuation token">:</SPAN>
        <SPAN class="keyword token">if</SPAN> row<SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">==</SPAN> None<SPAN class="punctuation token">:</SPAN>
            <SPAN class="keyword token">pass</SPAN>
        <SPAN class="keyword token">elif</SPAN> <SPAN class="string token">"\n"</SPAN> <SPAN class="keyword token">in</SPAN> row<SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">:</SPAN>
            <SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN>row<SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token">###########################################</SPAN>
<SPAN class="comment token"># this fails Search cursor to text file for output in Spyder console or ArcMap Python Window:</SPAN>
<SPAN class="comment token"># -*- coding: utf-8 -*-        </SPAN>
<SPAN class="keyword token">import</SPAN> arcpy<SPAN class="punctuation token">,</SPAN>sys
<SPAN class="comment token">#arcpy.env.workspace = r'I:\GIS\ArcSDE\SuperUser\pwengfc\SLCOen@pweng.sde' #r'J:\WaterQuality\test_tables.gdb'</SPAN>
arcpy<SPAN class="punctuation token">.</SPAN>env<SPAN class="punctuation token">.</SPAN>workspace <SPAN class="operator token">=</SPAN> r<SPAN class="string token">'J:\WaterQuality\test_tables.gdb'</SPAN>
<SPAN class="comment token">#tableList = ['BacteriaSamples', 'BacteriaBatch', 'BacteriaSamples', 'FieldParameters', 'MacrosSamples']</SPAN>
tableList <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="string token">'MacrosSamples'</SPAN><SPAN class="punctuation token">]</SPAN>

logFile <SPAN class="operator token">=</SPAN> r<SPAN class="string token">'C:\scripts\logs\newlines3.txt'</SPAN>
logoutput <SPAN class="operator token">=</SPAN> open<SPAN class="punctuation token">(</SPAN>logFile<SPAN class="punctuation token">,</SPAN><SPAN class="string token">'w'</SPAN><SPAN class="punctuation token">)</SPAN>
sys<SPAN class="punctuation token">.</SPAN>stdout <SPAN class="operator token">=</SPAN> logoutput


<SPAN class="keyword token">for</SPAN> table <SPAN class="keyword token">in</SPAN> tableList<SPAN class="punctuation token">:</SPAN>
    fieldNames <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">]</SPAN>
    tableFields <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>ListFields<SPAN class="punctuation token">(</SPAN>table<SPAN class="punctuation token">,</SPAN><SPAN class="string token">''</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'String'</SPAN><SPAN class="punctuation token">)</SPAN>
    <SPAN class="keyword token">for</SPAN> f <SPAN class="keyword token">in</SPAN> tableFields<SPAN class="punctuation token">:</SPAN>
        fieldNames<SPAN class="punctuation token">.</SPAN>append<SPAN class="punctuation token">(</SPAN>f<SPAN class="punctuation token">.</SPAN>name<SPAN class="punctuation token">)</SPAN>      
    <SPAN class="keyword token">for</SPAN> fieldName <SPAN class="keyword token">in</SPAN> fieldNames<SPAN class="punctuation token">:</SPAN>
        <SPAN class="comment token">#print('{} = table; {} = field name'.format(table,fieldName))</SPAN>
        field <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN>fieldName<SPAN class="punctuation token">]</SPAN>
        <SPAN class="keyword token">with</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>SearchCursor<SPAN class="punctuation token">(</SPAN>table<SPAN class="punctuation token">,</SPAN>field<SPAN class="punctuation token">)</SPAN><SPAN class="keyword token">as</SPAN> cursor<SPAN class="punctuation token">:</SPAN>
            <SPAN class="keyword token">for</SPAN> row <SPAN class="keyword token">in</SPAN> cursor<SPAN class="punctuation token">:</SPAN>
                <SPAN class="keyword token">if</SPAN> row<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">==</SPAN> None<SPAN class="punctuation token">:</SPAN>
                    <SPAN class="keyword token">pass</SPAN>
                <SPAN class="keyword token">elif</SPAN> <SPAN class="string token">"\n"</SPAN> <SPAN class="keyword token">in</SPAN> row<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">:</SPAN>
                    <SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">'{} = tablename; {} = fieldvalue'</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN>table<SPAN class="punctuation token">,</SPAN>row<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token">######################</SPAN>
<SPAN class="comment token"># and this fails  Update Cursor  in Spyder console or ArcMap Python Window::</SPAN>
<SPAN class="comment token"># -*- coding: utf-8 -*- </SPAN>
<SPAN class="keyword token">import</SPAN> arcpy
<SPAN class="comment token">#tableList = ['BacteriaBatch', 'BacteriaSamples', 'FieldParameters', 'MacrosSamples']</SPAN>
arcpy<SPAN class="punctuation token">.</SPAN>env<SPAN class="punctuation token">.</SPAN>workspace <SPAN class="operator token">=</SPAN> r<SPAN class="string token">'J:\WaterQuality\test_tables.gdb'</SPAN>
ws <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>env<SPAN class="punctuation token">.</SPAN>workspace

edit <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>Editor<SPAN class="punctuation token">(</SPAN>ws<SPAN class="punctuation token">)</SPAN>
edit<SPAN class="punctuation token">.</SPAN>startEditing<SPAN class="punctuation token">(</SPAN><SPAN class="token boolean">False</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="token boolean">False</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="comment token">#use False False: with undo,(False) multiuser mode(False)</SPAN>
edit<SPAN class="punctuation token">.</SPAN>startOperation<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>
tableList <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="string token">'MacrosSamples'</SPAN><SPAN class="punctuation token">]</SPAN>
<SPAN class="keyword token">for</SPAN> table <SPAN class="keyword token">in</SPAN> tableList<SPAN class="punctuation token">:</SPAN>
    fieldNames <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">]</SPAN>
    tableFields <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>ListFields<SPAN class="punctuation token">(</SPAN>table<SPAN class="punctuation token">,</SPAN><SPAN class="string token">''</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'String'</SPAN><SPAN class="punctuation token">)</SPAN>
    <SPAN class="keyword token">for</SPAN> f <SPAN class="keyword token">in</SPAN> tableFields<SPAN class="punctuation token">:</SPAN>
        fieldNames<SPAN class="punctuation token">.</SPAN>append<SPAN class="punctuation token">(</SPAN>f<SPAN class="punctuation token">.</SPAN>name<SPAN class="punctuation token">)</SPAN>
        <SPAN class="keyword token">for</SPAN> fieldName <SPAN class="keyword token">in</SPAN> fieldNames<SPAN class="punctuation token">:</SPAN>
            field <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN>fieldName<SPAN class="punctuation token">]</SPAN>
        <SPAN class="keyword token">with</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>da<SPAN class="punctuation token">.</SPAN>UpdateCursor<SPAN class="punctuation token">(</SPAN>table<SPAN class="punctuation token">,</SPAN>field<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> updateRows<SPAN class="punctuation token">:</SPAN>
            <SPAN class="keyword token">for</SPAN> updateRow <SPAN class="keyword token">in</SPAN> updateRows<SPAN class="punctuation token">:</SPAN>
                <SPAN class="keyword token">if</SPAN> updateRow<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="operator token">==</SPAN> None<SPAN class="punctuation token">:</SPAN>
                    <SPAN class="keyword token">pass</SPAN>
                <SPAN class="keyword token">elif</SPAN> <SPAN class="string token">"\n"</SPAN> <SPAN class="keyword token">in</SPAN> updateRow<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">:</SPAN>
                    updateRow<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> updateRow<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>replace<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"\n"</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">"=>"</SPAN><SPAN class="punctuation token">)</SPAN>
                    updateRows<SPAN class="punctuation token">.</SPAN>updateRow<SPAN class="punctuation token">(</SPAN>updateRow<SPAN class="punctuation token">)</SPAN>
                    <SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Updated {}  {}'</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN>table<SPAN class="punctuation token">,</SPAN>field<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>    
edit<SPAN class="punctuation token">.</SPAN>stopOperation<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>
edit<SPAN class="punctuation token">.</SPAN>stopEditing<SPAN class="punctuation token">(</SPAN><SPAN class="token boolean">True</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token">########################‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

DanPatterson_Retired

Joe What is the origin of the tables you are working with? not dbase? are the gdb tables?

bixb0012

ArcPy Cursors return Unicode, even in ArcMap, so it isn't the data or the cursor. What code is generating this error message?

JoeBorgione

It's embedded in a field value. I don't 'see' it at all; that's the problem. From last week's exercise, I'm plowing through a list of tables, and for each text field I replace the \n with 'real' text character. My loop and update cursor work splendidly, until it gets to one field in the last table of the list....

DanPatterson_Retired

Joe,

what is the string? (post it if you can at least the first 12 or so characters

How are you 'getting it/reading it'

Sadly, if these things are embedded, sometimes they are hard to trap before you can encode/decode them

Připojte se k diskusi

Přihlášení členové mohou přispívat, sledovat aktualizace a další. Jste tu noví? Zaregistrujte si bezplatný účet.

Rychlé odkazy

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.