gdal.VectorTranslate() transforms non-ASCII characters?

Question

I'm trying a workflow using gdal.VectorTranslate(), since ogr2ogr isn't there anymore.

I'm having an issue of the original data using non-ASCII characters, but they are replaced by � when translated.

How can I make this not happen?

I tried setting the config options, but "OGR_FORCE_ASCII" is only used by certain drivers and processes. Similarly, I cannot find a Translate option that would appear to take care of this.

This is kind of a major thing. If I absolutely have to, I suppose I can get the strings from the original data, but that will majorly slow things down, not to mention complicate things. 
Thanks

Edit it appears that this is dependent on the source; I have no problems going from fGDB to fGDB, but the real data is in an MDB.

AlfredBaldenweck · Answer

It's ANSI, it appears.

I'm looking at finding all text fields, cycling through them, and then going through with an update cursor on the final product to update them to the correct values. Not 100% on how I'm going to do all that (for reasons I don't really want to get into I had to create a new unique ID field for each table during the Translate() process), but we're going to try. It'd be fine if they just brought over the values as-is and Pro couldn't read them, but they evaluate them for utf-8, freak out, and then change the values.

Hayden Welch · Answer

https://github.com/OSGeo/gdal/blob/9d2c301cb3e18d2fea3af32652d0a31de0447e10/apps/ogr2ogr_lib.cpp#L90

https://gdal.org/en/stable/doxygen/classCPLStringList.html

Seems like they're using a char array for strings? Could be that the source data isn't properly encoded, or is encoded as something that isn't utf-8 (latin1? cp1252?)

You're really having a lot of fun issues with encoding lately huh

AlfredBaldenweck · Answer

Okay, with the help of this post  I'm able to read it alright.

import codecs
inmdb = ogr.Open(in_ds)
sql = "Select TextString from LDAnno"
res = inmdb.ExecuteSQL(sql)
for r in res:
    t = r.GetFieldAsBinary(0).hex()
    t = codecs.decode(t, "hex").decode('dbcs')
    print(t)

S½NE¼ W½SE¼ E½SW¼ E½SW¼ W½NE¼ W½NE¼ S½NW¼ N½SW¼

The question remains of how do I force gdal to use that reading instead of doing its own thing?