topic Re: UnicodeDecodeError: utf8 codec can't decode byte invalid continuation byte in Python Questions

Re: UnicodeDecodeError: utf8 codec can't decode byte invalid continuation byte

Yaron_YosefCohen — Sat, 11 Dec 2021 21:20:46 GMT

From: SearchCursor directory and subdirectories using python

I run the code and py fined layers with YEUD=20

but i also get en error:

Traceback (most recent call last):
  File "C:\Users\yaron.KAYAMOT\Desktop\geonet.PY", line 11, in <module>
    for row in rows:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe7 in position 23: invalid continuation byte
>>>

i use files and folders names Right to left- how can i solve this issue please?

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

curtvprice — Sun, 17 May 2015 10:35:02 GMT

I started a new thread on this issue.

These links may help, but I have no experience with international codesets -- perhaps you need to encode your path strings before parsing them, or do something with locale - or somehow change your codec to Latin-1?

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 6: ordinal not in range(128)

python - UnicodeDecodeError, invalid continuation byte - Stack Overflow

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

RebeccaStrauch__GISP — Sun, 17 May 2015 16:21:41 GMT

Check to make sure your indentations are correct and : are placed where they should be, etc. I've received similar errors when I've had just a typo several lines up. i know this question was spun off another with coding suggestions., so maybe something was missed when merging with your code. Errors can be deceiving sometimes, so worth a syntax check.

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

Yaron_YosefCohen — Mon, 18 May 2015 04:28:06 GMT

Hi Rebecca,

i run the code and there where no identations error-else python would not run the code

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

RebeccaStrauch__GISP — Mon, 18 May 2015 14:52:31 GMT

YYC - It may be helpful for others if you post the full code where you are getting an error. The post this was spun off of shows Curtis' suggestion, but not you final code.

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

Yaron_YosefCohen — Sat, 11 Dec 2021 21:20:49 GMT

Hi Rebecca,

this is the code:

import os
import fnmatch
import arcpy
rootPath = r"C:\Project\layers" 
pattern = 'mig*.shp' 
for root, dirs, files in os.walk(rootPath): 
    for filename in fnmatch.filter(files, pattern): 
        shp = os.path.join(root, filename)
        if arcpy.ListFields(shp, "YEUD"):
            print("{} has YEUD field".format(shp)) 
            with arcpy.da.SearchCursor(shp, ["YEUD"]) as rows:
                for row in rows:
                    if row[0] == 20:
                        print("{} has a record with YEUD = 20".format(shp))
                        break

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

curtvprice — Tue, 19 May 2015 05:01:53 GMT

Based on this:

character encoding - Python: Converting from ISO-8859-1/latin1 to UTF-8 - Stack Overflow

I'm wondering if you may be able to fix this by doing a quick conversion to utf-8:

shp = (os.path.join(root, filename)).encode("utf-8")

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

Yaron_YosefCohen — Sat, 11 Dec 2021 21:20:51 GMT

i get an error :

>>> 
Traceback (most recent call last):
  File "C:\Users\yaron.KAYAMOT\Desktop\python.py", line 9, in <module>
    shp = (os.path.join(root, filename)).encode("utf-8")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe7 in position 23: ordinal not in range(128)
>>>

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

curtvprice — Tue, 19 May 2015 05:44:29 GMT

Sorry, I didn't read the post I linked carefully enough. In this example iso-8859-1 is the codes your paths are in.

shp.decode('iso-8859-1').encode('utf-8')

I'm going to link the post again because it has lots of really useful information about unicode, latin1, utf-8...

python - UnicodeDecodeError, invalid continuation byte - Stack Overflow

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

Yaron_YosefCohen — Tue, 19 May 2015 05:52:16 GMT

where to insert this route?

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

curtvprice — Tue, 19 May 2015 05:58:44 GMT

This line is a replacement for line 08. I was hoping someone that knows more about international characters will chime in!

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

Yaron_YosefCohen — Sat, 11 Dec 2021 21:20:54 GMT

i get an error:

>>>


Traceback (most recent call last):
  File "C:\Users\yaron.KAYAMOT\Desktop\python.py", line 9, in <module>
    shp.decode('iso-8859-1').encode('utf-8')
NameError: name 'shp' is not defined
>>>

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

curtvprice — Tue, 19 May 2015 06:08:28 GMT

Sorry:

shp = (os.path.join(root, filename)).decode('iso-8859-1').encode("utf-8")

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

Yaron_YosefCohen — Tue, 19 May 2015 06:16:53 GMT

an error:

IOError: <unprintable IOError object>

i think now there no unicode error -but there problem else

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

curtvprice — Tue, 19 May 2015 06:25:30 GMT

I think you're going to have to find someone that has a better handle on this unicode stuff. I added some tags that maybe will attract some help.

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

RebeccaStrauch__GISP — Sat, 11 Dec 2021 21:20:57 GMT

YYC,

I'm not familiar with unicode, but I did test you code with a some of my own shape files against a field ACRES (format DOUBLE) and it worded without an error. What is the format of your "YEUD" field? The code I used (changed to match my data):

import os  
import fnmatch  
import arcpy  
rootPath = r"D:\Project\layers"   
pattern = 'lake*.shp'   
for root, dirs, files in os.walk(rootPath):   
    for filename in fnmatch.filter(files, pattern):   
        shp = os.path.join(root, filename)  
        if arcpy.ListFields(shp, "ACRES"):  
            print("{} has ACRES field".format(shp))   
            with arcpy.da.SearchCursor(shp, ["ACRES"]) as rows:  
                for row in rows:  
                    if row[0] > 300000:  
                        print("{} has a record with ACRES > 3000000".format(shp))  
                        break

My results were:

D:\Project\layers\lakes.shp has ACRES field

D:\Project\layers\lakes.shp has a record with ACRES > 3000000

D:\Project\layers\lake_10.shp has ACRES field

D:\Project\layers\lake_11.shp has ACRES field

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

curtvprice — Tue, 19 May 2015 18:30:09 GMT

YYC's issue is not my scripting -- he's working with international characters in the path names, which are way out of my expertise!! The YEUD field is an integer (in the international language of IEEE representations) so there is not a problem there.

YYC, I'm afraid think you may need to ask for tech support from your local Esri affiliate or user groups on this one.

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

Yaron_YosefCohen — Wed, 20 May 2015 04:57:51 GMT

Rebecca,

"YEUD" field type is Double

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

curtvprice — Wed, 20 May 2015 05:08:11 GMT

As I said YEUD is not the issue, it's the feature class pathname (shp) using those funky right-to-left characters.

Re: UnicodeDecodeError: utf8' codec can't decode byte invalid continuation byte

RebeccaStrauch__GISP — Wed, 20 May 2015 05:11:10 GMT

I would gave to agree with Curtus then, if the code i show above doesn't work on the YEUD field then it's probably above my knowledge. So, sorry, but I can't offer anything more, hopefully someone else can help you out.

i did grab the code myself, not that I have an immediate need to do what you are trying to do, but it might be something I can use in the future. Nice little utility (at least with my data).