Sitelet https://web.archive.org/web/20210114134632/https://github.com/pallets/werkzeug/issues/1760
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

investigate removing get_filesystem_encoding #1760

Open
davidism opened this issue Mar 16, 2020 · 3 comments
Open

investigate removing get_filesystem_encoding #1760

davidism opened this issue Mar 16, 2020 · 3 comments

Comments

@davidism
Copy link
Member

@davidism davidism commented Mar 16, 2020 •

I think this is a non-issue in Python >= 3.6 because of PEP 529, but I haven't had time to look thoroughly. It would be helpful if someone could write up some research on whether it's still relevant.

@gmelodie
Copy link

@gmelodie gmelodie commented Jun 24, 2020 •

So there are mainly two things that get_filesystem_encoding checks:

  1. Whether or not the system is a "buggy" unicode filesystem, i.e. linux or bsd system and the result of sys.getfilesystemencoding is None (which can't be anymore so this check should no longer make sense)
  2. Whether or not sys.getfilesystemencoding is ASCII

In theory has_likely_buggy_unicode_filesystem should no longer be needed as sys.getfilesystemencoding will never return None, but according to the python docs:

On Unix, the encoding is the locale encoding.

And according to the werkzeug docs:

several bug reports against Werkzeug have shown that the value of sys.getfilesystemencoding() cannot be trusted under traditional UNIX systems. The usual problems come from misconfigured systems, where LANG and similar environment variables are not set

So I think we shouldn't be able to safely remove get_filesystem_encoding. This is all theoretical thinking based on docs as I am unsure about how to create tests for this.

Edit: It seems that using python >=3.7's UTF-8 mode could solve the issue, but it still has to be explicitly specified

@davidism
Copy link
Member Author

@davidism davidism commented Jun 25, 2020

Great writeup, thank you. It sounds like we can at least clean this up a bit, even if we can't remove it yet. The comment in the Werkzeug docs is pretty vague, were you able to find any of the issues it referred to?

@gmelodie
Copy link

@gmelodie gmelodie commented Jun 26, 2020

Here's what comes up searching for getfilesystem on the repo. From these, #689 and #635 seem to be relevant and have been fixed by #674, which implements the get_filesystem_encoding. I think we could try and create some tests for this and see exactly what can be removed from the code without issues. I'm not sure about how to implement these tests for different architectures though, but it should be fun :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.