Commit cea16735 authored by Zeyi (Rice) Fan's avatar Zeyi (Rice) Fan Committed by Facebook Github Bot

fix encoding issue when building on Windows

Summary:
On Windows, the writing operation would fail with:

```
Traceback (most recent call last):
  File ".\opensource\fbcode_builder\getdeps.py", line 400, in <module>
    sys.exit(main())
  File ".\opensource\fbcode_builder\getdeps.py", line 393, in main
    return args.func(args)
  File ".\opensource\fbcode_builder\getdeps.py", line 236, in run
    change_status = fetcher.update()
  File "C:\open\fbsource\fbcode\opensource\fbcode_builder\getdeps\fetcher.py", line 451, in update
    return mapping.mirror(self.build_options.fbsource_dir, self.repo_dir)
  File "C:\open\fbsource\fbcode\opensource\fbcode_builder\getdeps\fetcher.py", line 400, in mirror
    f.write(name + "\n")
  File "C:\Python36\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 104-105: character maps to <undefined>
```

and this is caused by a file in libgit2: https://github.com/libgit2/libgit2/blob/master/tests/resources/status/%E8%BF%99, which is intended to test handling non-ASCII path.

Python on Windows will write file in cp1252 encoding by default, which does not contain that Chines character. (Caveat: that file on my system doesn't have the correct file name as well, it is being encoded in IBM861 for some reason. However the characters in IBM861 does not exist in CP1252 either)

Reviewed By: wez

Differential Revision: D15281521

fbshipit-source-id: 8a75e32bc1042167c945d67e26b549fda83b6b41
parent 4be1b74a
......@@ -388,17 +388,17 @@ class ShipitPathMap(object):
# may legitimately need to keep some state in the source tree :-/
installed_name = os.path.join(dest_root, ".shipit_shipped")
if os.path.exists(installed_name):
with open(installed_name, "r") as f:
for name in f.readlines():
with open(installed_name, "rb") as f:
for name in f.read().decode("utf-8").splitlines():
name = name.strip()
if name not in full_file_list:
print("Remove %s" % name)
os.unlink(name)
change_status.record_change(name)
with open(installed_name, "w") as f:
with open(installed_name, "wb") as f:
for name in sorted(list(full_file_list)):
f.write("%s\n" % name)
f.write(("%s\n" % name).encode("utf-8"))
return change_status
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment