ArcPy fails due to Unicode encoding error

519
5
Jump to solution
11-27-2023 09:51 AM
Labels (1)
rbd
by
New Contributor II

Hi All,

I'm automating exports of some GDB feature class attribute tables to CSV using Python. One FC works beautifully and the other produces this error:

UnicodeEncodeError                        Traceback (most recent call last)
In  [27]:
Line 9:     csvwriter.writerow(row)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\encodings\cp1252.py, in encode:
Line 19:    return codecs.charmap_encode(input,self.errors,encoding_table)[0]

UnicodeEncodeError: 'charmap' codec can't encode character '\u0374' in position 149: character maps to <undefined>

Does anyone know what I should do to fix this?

-r

0 Kudos
1 Solution

Accepted Solutions
DavidSolari
Occasional Contributor III

You open the file on line 27, try adding

encoding="utf-8"

as parameter and see if that fixes things.

View solution in original post

0 Kudos
5 Replies
DavidSolari
Occasional Contributor III

Based on the encoding the CSV writer pulled in you're trying to write the data to a file that isn't Unicode compatible. Specify a suitable encoding when you open the file ("utf-8" works in virtually every case) and you should make more progress.

0 Kudos
rbd
by
New Contributor II

Thanks David. I've looked at a number of resources for troubleshooting this, and have read that utf-8 is used, but I have not been successful at placing that specification in my code in a way that works. Where in the code should this be specified? Below is my code, up to the point of failure:

# import the modules, including ones that hopefully troubleshoot the unicode encoding error
import arcpy
from arcpy import da
import csv
import codecs
import os
import fileinput
import sys
# set the workspace environments
# file location
arcpy.env.workspace = r"C:workspace"
# overwrite files of the same name in the output location
arcpy.env.overwriteOutput = True
# transfer the domains and descriptions to shapefile headers and attributes
arcpy.env.transferDomains = True
#define the input feature classes output file names and types as variables
in_gdb = r"C:In.gdb"
in_sites = f"{in_gdb}/WSPRC_CulturalResources/ResourcePolyRecordedUpdated"
sites_shp = f"{arcpy.env.workspace}/ResourcePolyRecordedUpdated.shp"
out_sites_temp = f"{arcpy.env.workspace}/DoNotModify_CurrentSitesListTemp.csv"
out_sites = f"{arcpy.env.workspace}/DoNotModify_CurrentSitesList.csv"
## export the feature classes to shapefiles
arcpy.conversion.FeatureClassToShapefile(in_sites, arcpy.env.workspace)
# delete the unnecessary fields from the exported shapefiles
arcpy.management.DeleteField(sites_shp, ["Shape_Leng", "Shape_Area"])
#write the csv
with open(out_sites_temp, "w") as csvfile:
    csvwriter = csv.writer(csvfile, delimiter=',', lineterminator='\n')
    ## Write field name header line
    fields = ['ProjName', 'Date', 'Acreage', 'Notes']
    csvwriter.writerow(fields), 
    ## Write data rows
    with arcpy.da.SearchCursor(sites_shp, fields) as s_cursor:
        for row in s_cursor:
            csvwriter.writerow(row)

 

0 Kudos
DavidSolari
Occasional Contributor III

You open the file on line 27, try adding

encoding="utf-8"

as parameter and see if that fixes things.

0 Kudos
rbd
by
New Contributor II

Success @DavidSolari ! I had tried that piece last week in the row below, where other parameters seem to be listed. Do you know the reason it should go in Line 27 rather than Line 28? I'm trying to understand the inner workings of the code so I can improve. Thanks again!

0 Kudos
DavidSolari
Occasional Contributor III

It might be worth a quick trip to the official Python docs for file objects. In short, the "open" function returns a file object with various settings from the function, including the file's encoding. The "writer" object from the "csv" module is just a wrapper around the file object that translates raw CSV data to and from Python data types; you can write a CSV file without the wrapper but it makes it easier.

0 Kudos