topic Re: Unicode error in Python Questions

Unicode error

AmyKlug — Sun, 12 Dec 2021 01:52:02 GMT

Hi,

When i am running this code i am getting a Unicode error half way through. Not sure what mxd/layer file name is hanging it up (line 33) on but not sure where to put the fix (line 7 - or if this is the correct fix) either.

import arcpy, os

#code adds mxd name and layer path name to text file separated by a comma
arcpy.env.overwriteOutput = True


#def Utf8EncodeArray(oldArray):
    #newArray = []
    #for element in oldArray:
        #if isinstance(element, unicode):
        #newArray.append(element.encode("utf-8"))
    #else:
        #newArray.append(element)
    #return newArray


path = "////serverpath"
#path2 =
mxdlst = []
txt = open("text file path", 'w')
print "making mxd list"
for root, dirs, files in os.walk(path):
    for fname in files:
        if fname.endswith(".mxd"):
            mxd = root + '\\' + fname
            mxdlst.append(mxd)
del mxd, fname
for mapdoc in mxdlst:
    mxd = arcpy.mapping.MapDocument(mapdoc)
    for df in arcpy.mapping.ListDataFrames(mxd, "*"):
        for lyrlst in arcpy.mapping.ListLayers(mxd, "*", df):
            if lyrlst.supports("DATASOURCE"):
                txt.write(mapdoc + "," + lyrlst.workspacePath + "\\" + lyrlst.name + "\n")
                print "adding" + mapdoc + "," + lyrlst.workspacePath + "\\" + lyrlst.name + "\n"
            else:
                txt.write(mapdoc + "," + lyrlst.name + "\n")
                print "adding" + mapdoc + "," + lyrlst.name + "\n"
txt.close()
del mxd, df, lyrlst, mapdoc, mxdlst

Re: Unicode error

DanPatterson_Retired — Wed, 29 Jul 2015 22:34:08 GMT

What is the layer name etc? If it contains characters that need to be converted, then you will have to do so

Re: Unicode error

AmyKlug — Wed, 29 Jul 2015 23:06:51 GMT

I need code to check for that and fix it just not sure where to put it

Re: Unicode error

DanPatterson_Retired — Wed, 29 Jul 2015 23:53:50 GMT

haven't had the unicode issues yet, but apparently, I will have to and one suggestion is to specify encoding at the top of the script with

# -*- coding: utf-8 -*-

but some unicode types should step up since it appears that you have a character that can't be represented by the ASCII chars in the range 0-127

Re: Unicode error

AmyKlug — Thu, 30 Jul 2015 00:04:57 GMT

I have seen that before. does it work with the # sign in front?

Re: Unicode error

DanPatterson_Retired — Thu, 30 Jul 2015 00:23:31 GMT

apparently that is what is supposed be done, first line. but I seriously haven't played with encoding other than ascii ... I really should given accented characters etc, but so far, haven't had to deal with. My only suggestion is find someone that works with such data and or look at at one of the files and see what characters are present there. Sorry I can't help more, but searching on your error message on GeoNet may turn up more

Re: Unicode error

XanderBakker — Sun, 12 Dec 2021 01:52:05 GMT

An interesting article to read would be:

Solving Unicode Problems in Python 2.7 | Azavea Labs

And to give an example of what works and fails:

# -*- coding: utf-8 -*-
myText = u"example of únì¢ødë"

# this works:
print myText

# UnicodeEncodeError: 'ascii' codec can't encode character u'\xfa' in position 11: ordinal not in range(128)
print "{0}".format(myText)
# inserting unicode in a str

# This works 
print u"{0}".format(myText)
# inserting unicode into a unicode

# UnicodeEncodeError: 'ascii' codec can't encode character u'\xfa' in position 11: ordinal not in range(128)
print u"{0}".format(myText.decode('utf-8'))
# inserting utf-8 into unicode

Re: Unicode error

DanPatterson_Retired — Sun, 12 Dec 2021 01:52:08 GMT

what a kludge...

# -*- coding: utf-8 -*-
chars = [unichr(i) for i in range(0,256) if (32 < i < 128) or (i > 161)]
print "Unicode characters 33-127 and 161-255\n" + ("{:5}"*len(chars)).format(*chars)

Unicode characters 33-127 and 161-255

! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b c d e f g h i j k l m n o p q r s t u v w x y z { | } ~ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ® ¯ ° ± ² ³ ´ µ ¶ · ¸ ¹ º » ¼ ½ ¾ ¿ À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï Ð Ñ Ò Ó Ô Õ Ö × Ø Ù Ú Û Ü Ý Þ ß à á â ã ä å æ ç è é ê ë ì í î ï ð ñ ò ó ô õ ö ÷ ø ù ú û ü ý þ ÿ

anything not visible, isn't...can't wait to learn all of them

Re: Unicode error

Luke_Pinner — Sun, 12 Dec 2021 01:52:11 GMT

Dan Patterson:
...one suggestion is to specify encoding at the top of the script with # -*- coding: utf-8 -*-

This only applies to literal characters in the python script file itself, not any string variables when the script is run.

test_noenc.py

somestr = u'über còól'
print somestr

test_enc.py

# -*- coding: utf-8 -*-
somestr = u'über còól'
print somestr

C:\Temp>python test_noenc.py
File "test_noenc.py", line 1
SyntaxError: Non-ASCII character '\xfc' in file test_noenc.py on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

C:\Temp>python test_enc.py
über còól

Re: Unicode error

DanPatterson_Retired — Fri, 31 Jul 2015 06:17:39 GMT

Thanks Luke...fixed

Re: Unicode error

AmyKlug — Fri, 31 Jul 2015 17:03:10 GMT

is this how to do it?

txt.write(u'mapdoc' + "," + u'lyrlst.workspacePath' + "\\" + u'lyrlst.name' + "\n")

Re: Unicode error

XanderBakker — Sun, 12 Dec 2021 01:52:13 GMT

This will not be the solution due to a number of reasons:

If you use something like u'lyrlst.workspacePath', then you create a unicode text contains the text lyrlst.workspacePath and not the content of the variable. For that you would have to use unicode(lyrlst.workspacePath).

The other aspect is combining a normal str with a unicode. In your line there are still some elements that are of type str ("," "\\", "\n", etc).

I have seen posts where they recommend using the codecs library to write unicode to text file:

import codecs
with codecs.open(YourOutputFile, 'w', encoding='utf-8') as f:
    f.write(unicode_text)

Please take the time to read this article: Solving Unicode Problems in Python 2.7 | Azavea Labs It can help to clear thing up for you.