nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

ArcPy Encoding Problem

Nicole_Ueberschär

Hello folks,

I have to work with txt files that contain special (non-ascii) characters like in "Temperature (°C)" or "Chl_A (µg/L)".

When I read the field names directly from the table with utf-8 encoding and list them in Python I get 'Temperature (\xc2\xb0C)' and 'Chl_A (\xb5g/L)'.

I receive the names of the fields to work on from a script tool where the validation tool reads the field names (with utf-8 encoding) and gives them back as a list to chose from (where they look "properly spelled"). These chosen ones are then received from my python script with GetParameterAsText and then split up.

fieldnames<SPAN class="operator token">=</SPAN>arcpy<SPAN class="punctuation token">.</SPAN>GetParameterAsText<SPAN class="punctuation token">(</SPAN><SPAN class="number token">3</SPAN><SPAN class="punctuation token">)</SPAN>
fieldnameslist<SPAN class="operator token">=</SPAN>fieldnames<SPAN class="punctuation token">.</SPAN>split<SPAN class="punctuation token">(</SPAN><SPAN class="string token">";"</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

When I read and write those I get [u"'Chl_A (\xb5g/L)'", u"'Temperature (\xb0C)'"]

Note also the difference between the first and the second output: (\xc2\xb0C) and (\xb0C) which seem to be two different unicode encodings. When I read them again in a new list with utf-8 encoding I get ["'Chl_A (\xc2\xb5g/L)'", "'Temperature (\xc2\xb0C)'"]

Now when I try to use the field name again to read from the table the values I need for further calculations the field name is not recognised and I guess that is because of the encoding. Interestingly (at least for me), when I read from the utf-8 list or from the other the field name I get again "Temperature (°C)" or "Chl_A (µg/L)" as an output. So I assume I have to "translate" the symbols back into hexa spelling to be able to communicate with the txt file.

When I hardcode the field name to be 'Chl_A (\xb5g/L)' in the arcpy.message I still get Chl_A (µg/L) but then the search cursor complains that "'utf8' codec can't decode byte 0xb5 in position 7: invalid start byte". When I use u'Chl_A (\xb5g/L)' the field name is recognized.

How can I give the field names from my list so that the script can read them and they are recognized in the table again for the search cursor?

I'm using Python 2.7.14 with ArcMap 10.6.1

PS: Looking for further information I tumbled over this read, which might give you something to laugh for today: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excu… (plus some well explained information I think, but it didn't help solve my problem...).

Find more posts tagged with

Accepted answers

RandyBurton

Thanks for attaching the files. From that I noticed, you wanted the option to select multiple fields. I modified the validator code in your other post for this option.

I believe that the encoding issues arise because the tool interface appears to pass values that contain special characters in single quotes. From the tool printout, note line 7 below. The two fields with special characters are enclosed within single quotes and inside double quotes.

Executing: encodingtest C:\path\to\folder\encoding 'Chl_A (µg/L)';'Temperature (°C)';pH
Start Time: Sun Feb 24 19:55:53 2019
Running script encodingtest...

Table view field names: ['Original_file_&_sheet', 'Campaign', 'Profile_No', 'Date', 'Longitude (degrees_east)', 'Latitude (degrees_north)', 'Depth (m)', 'Temperature (\xc2\xb0C)', 'pH', 'Chl_A (\xc2\xb5g/L)', 'UID', 'UIDGraph']

fieldnamesliste from parameter: ["'Chl_A (\xc2\xb5g/L)'", "'Temperature (\xc2\xb0C)'", 'pH']‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

By trimming the quotes, I was able to get your script to work. Here's the section I modified; it starts around line 142 in the original code. The .decode("utf-8") in line 4 below may be the result of some of the encoding/decoding you were doing earlier in the script. I'm not sure what of that code can safely be removed.

<SPAN class="keyword token">def</SPAN> <SPAN class="token function">PrintPlot</SPAN><SPAN class="punctuation token">(</SPAN>gs<SPAN class="punctuation token">,</SPAN> gsNr<SPAN class="punctuation token">,</SPAN> table<SPAN class="punctuation token">,</SPAN> fieldname<SPAN class="punctuation token">,</SPAN> min<SPAN class="punctuation token">,</SPAN> max<SPAN class="punctuation token">,</SPAN> xlabelname<SPAN class="punctuation token">,</SPAN> colorL<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
    <SPAN class="keyword token">if</SPAN> fieldname<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">==</SPAN> <SPAN class="string token">"'"</SPAN><SPAN class="punctuation token">:</SPAN>
        arcpy<SPAN class="punctuation token">.</SPAN>AddMessage<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"apostrophe found"</SPAN><SPAN class="punctuation token">)</SPAN>
        fieldname <SPAN class="operator token">=</SPAN> fieldname<SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">:</SPAN><SPAN class="operator token">-</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>decode<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"utf-8"</SPAN><SPAN class="punctuation token">)</SPAN>
        xlabelname <SPAN class="operator token">=</SPAN> fieldname
    fields <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN>fieldname<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"Depth (m)"</SPAN><SPAN class="punctuation token">]</SPAN>‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Hope this helps.

All comments

Nicole_Ueberschär

Awesome, thanks a lot, Randy! Works great!

RandyBurton

Thanks for attaching the files. From that I noticed, you wanted the option to select multiple fields. I modified the validator code in your other post for this option.

Executing: encodingtest C:\path\to\folder\encoding 'Chl_A (µg/L)';'Temperature (°C)';pH
Start Time: Sun Feb 24 19:55:53 2019
Running script encodingtest...

Table view field names: ['Original_file_&_sheet', 'Campaign', 'Profile_No', 'Date', 'Longitude (degrees_east)', 'Latitude (degrees_north)', 'Depth (m)', 'Temperature (\xc2\xb0C)', 'pH', 'Chl_A (\xc2\xb5g/L)', 'UID', 'UIDGraph']

fieldnamesliste from parameter: ["'Chl_A (\xc2\xb5g/L)'", "'Temperature (\xc2\xb0C)'", 'pH']‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

<SPAN class="keyword token">def</SPAN> <SPAN class="token function">PrintPlot</SPAN><SPAN class="punctuation token">(</SPAN>gs<SPAN class="punctuation token">,</SPAN> gsNr<SPAN class="punctuation token">,</SPAN> table<SPAN class="punctuation token">,</SPAN> fieldname<SPAN class="punctuation token">,</SPAN> min<SPAN class="punctuation token">,</SPAN> max<SPAN class="punctuation token">,</SPAN> xlabelname<SPAN class="punctuation token">,</SPAN> colorL<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
    <SPAN class="keyword token">if</SPAN> fieldname<SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">==</SPAN> <SPAN class="string token">"'"</SPAN><SPAN class="punctuation token">:</SPAN>
        arcpy<SPAN class="punctuation token">.</SPAN>AddMessage<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"apostrophe found"</SPAN><SPAN class="punctuation token">)</SPAN>
        fieldname <SPAN class="operator token">=</SPAN> fieldname<SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">:</SPAN><SPAN class="operator token">-</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>decode<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"utf-8"</SPAN><SPAN class="punctuation token">)</SPAN>
        xlabelname <SPAN class="operator token">=</SPAN> fieldname
    fields <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN>fieldname<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"Depth (m)"</SPAN><SPAN class="punctuation token">]</SPAN>‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Hope this helps.

Nicole_Ueberschär

Attached is a zip file that contains a toolbox with a script tool, the corresponding script and a test file.

When you run the script tool it might throw the bad descriptor error again (as reported in my other posting) but you can just click ok and run the script. Sometimes you have to press ok twice.

Select as directory the directory where you saved the txt file (propably the main folder of the zip). I left you three parameters with a couple of values, two with special characters and one simple one.

In the script at lines 105 and 106 I write min and max values to a directory where I would like to have proper (readable) names, that's why I am using an extra variable (field_text, list is listfields_text) here. I need these values again from line 130 where I am creating a graph from the table values. Here I will need to loop through the field names again but cannot figure out how to reference them in a way it would be recognized in the table as column. In line 70 I found a work around which works as long as the parameters with special characters are limited (normally there are two more in the table) but it is not very elegant. I could do the same with the loop for the graphs but I feel like there must be a better way of dealing with these characters.

Thanks for looking into it!

encoding_test.zip

RandyBurton

Can you supply a small test file?

DanPatterson_Retired

I think it is the getparameterastext thing

gpat <SPAN class="operator token">=</SPAN> <SPAN class="string token">"Chl_A (\xb5g/L);Temperature (\xb0C)"</SPAN> <SPAN class="comment token"># GetParameterAsText ??????</SPAN>

<SPAN class="punctuation token">[</SPAN>i<SPAN class="punctuation token">.</SPAN>encode<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>decode<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">for</SPAN> i <SPAN class="keyword token">in</SPAN> gpat<SPAN class="punctuation token">.</SPAN>split<SPAN class="punctuation token">(</SPAN><SPAN class="string token">";"</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">]</SPAN>

<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'Chl_A (µg/L)'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'Temperature (°C)'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

But I don't know since everything reads fine in Python 3 since "string" is unicode

Nicole_Ueberschär

Hi Dan,

At which point would you suggest I need to do the decoding?

fieldnames<SPAN class="operator token">=</SPAN>arcpy<SPAN class="punctuation token">.</SPAN>GetParameterAsText<SPAN class="punctuation token">(</SPAN><SPAN class="number token">3</SPAN><SPAN class="punctuation token">)</SPAN>
fieldnamesliste<SPAN class="operator token">=</SPAN>fieldnames<SPAN class="punctuation token">.</SPAN>split<SPAN class="punctuation token">(</SPAN><SPAN class="string token">";"</SPAN><SPAN class="punctuation token">)</SPAN>
fieldnames_decoded<SPAN class="operator token">=</SPAN><SPAN class="punctuation token">[</SPAN>field<SPAN class="punctuation token">.</SPAN>encode<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"utf-8"</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>decode<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"utf-8"</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">for</SPAN> field <SPAN class="keyword token">in</SPAN> fieldnamesliste<SPAN class="punctuation token">]</SPAN>
fieldnames_encoded<SPAN class="operator token">=</SPAN><SPAN class="punctuation token">[</SPAN>field<SPAN class="punctuation token">.</SPAN>encode<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"utf-8"</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">for</SPAN> field <SPAN class="keyword token">in</SPAN> fieldnamesliste<SPAN class="punctuation token">]</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

From what I have seen until now I would assume I have somehow to use the fieldname including the u".." which I get when not en- and decoding or use encoding and decoding:

decoded: [u"'Chl_A (\xb5g/L)'"] (which is actually the same as not using encode+decode, which makes sense to me, I don't know why it doesn't print the u for you, but that might be Python3 depending)
encoded: ["'Chl_A (\xc2\xb5g/L)'"]

But when I read the field name again for using it for my search cursor I get for

decoded: field name: 'Chl_A (µg/L)'

encoded: field name: 'Chl_A (µg/L)'

Somehow I am stuck between this encoding and decoding. It seems like the python script automatically interpretes the hex codes...

DanPatterson_Retired

I don't have python 2.7 but in python 3.6.8

u"'Chl_A (\xb5g/L)'".encode().decode()

"'Chl_A (µg/L)'"

u"'Temperature (\xb0C)'".encode().decode()

"'Temperature (°C)'"<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.