I was just banging my head against a wall for a couple hours trying to figure out why all my text was being written back into my project using cp1252 encoding. Turns out that if you use the Path.open() method to read a file, it will defer to your system for text encoding. In the case of Windows, that is still cp1252 and not utf-8 even though all userspace text in Windows has been utf-8 for a few years. This was a nefarious bug since ASCII, cp1252, and UTF-8 are all interchangeable for English letters and punctuation. it wasn't until a field contained the • symbol and I got back • that I noticed.
Fix:
-- Path('<filepath>').open('wt').write(my_string)
++ Path('<filepath>').open('wt', encoding='utf-8').write(my_string)
Note:
Files will remember the encoding they were created in, so make sure you don't have any files containing unicode that are actually encoded as cp1252 or you'll be kicking yourself.
