• Zeyi (Rice) Fan's avatar
    fix encoding issue when building on Windows · cea16735
    Zeyi (Rice) Fan authored
    Summary:
    On Windows, the writing operation would fail with:
    
    ```
    Traceback (most recent call last):
      File ".\opensource\fbcode_builder\getdeps.py", line 400, in <module>
        sys.exit(main())
      File ".\opensource\fbcode_builder\getdeps.py", line 393, in main
        return args.func(args)
      File ".\opensource\fbcode_builder\getdeps.py", line 236, in run
        change_status = fetcher.update()
      File "C:\open\fbsource\fbcode\opensource\fbcode_builder\getdeps\fetcher.py", line 451, in update
        return mapping.mirror(self.build_options.fbsource_dir, self.repo_dir)
      File "C:\open\fbsource\fbcode\opensource\fbcode_builder\getdeps\fetcher.py", line 400, in mirror
        f.write(name + "\n")
      File "C:\Python36\lib\encodings\cp1252.py", line 19, in encode
        return codecs.charmap_encode(input,self.errors,encoding_table)[0]
    UnicodeEncodeError: 'charmap' codec can't encode characters in position 104-105: character maps to <undefined>
    ```
    
    and this is caused by a file in libgit2: https://github.com/libgit2/libgit2/blob/master/tests/resources/status/%E8%BF%99, which is intended to test handling non-ASCII path.
    
    Python on Windows will write file in cp1252 encoding by default, which does not contain that Chines character. (Caveat: that file on my system doesn't have the correct file name as well, it is being encoded in IBM861 for some reason. However the characters in IBM861 does not exist in CP1252 either)
    
    Reviewed By: wez
    
    Differential Revision: D15281521
    
    fbshipit-source-id: 8a75e32bc1042167c945d67e26b549fda83b6b41
    cea16735
fetcher.py 24.1 KB