Sitelet https://github.com/python/cpython/issues/78079
Skip to content

[pathlib] issues with Windows device paths #78079

Closed
Closed
[pathlib] issues with Windows device paths#78079
Labels
3.7 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifeOS-windowsstdlibStandard Library Python modules in the Lib/ directorytopic-pathlibtype-bugAn unexpected behavior, bug, or error

Description

@eryksun
eryksun
opened on Jun 18, 2018
BPO 33898
Nosy @pfmoore, @pitrou, @tjguk, @zware, @eryksun, @zooba
PRs
  • bpo-33898: Fix pathlib issues with Windows device paths #8671
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2018-06-18.22:40:29.969>
labels = ['type-bug', '3.8', '3.9', '3.7', 'library', 'OS-windows']
title = 'pathlib issues with Windows device paths'
updated_at = <Date 2020-04-17.17:59:50.770>
user = 'https://github.com/eryksun'

    bugs.python.org fields:

    activity = <Date 2020-04-17.17:59:50.770>
actor = 'pitrou'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2018-06-18.22:40:29.969>
creator = 'eryksun'
dependencies = []
files = []
hgrepos = []
issue_num = 33898
keywords = ['patch']
message_count = 2.0
messages = ['319921', '337405']
nosy_count = 6.0
nosy_names = ['paul.moore', 'pitrou', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower']
pr_nums = ['8671']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33898'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

    Linked PRs

    Activity

    1. eryksun commented on Jun 18, 2018

      @eryksun
      eryksun
      on Jun 18, 2018
      ContributorAuthor

      For \\?\ extended device paths, pathlib removes the trailing slash for the root directory if the device isn't UNC or a logical (A-Z) drive.

      Correct:

          >>> str(Path('//?/UNC/'))
    '\\\\?\\UNC\\'

    >>> str(Path('//?/Z:/'))
    '\\\\?\\Z:\\'

      Incorrect:

          >>> str(Path('//?/BootPartition/'))
    '\\\\?\\BootPartition'

    >>> str(Path('//?/Volume{}/'))
    '\\\\?\\Volume{}'

    >>> str(Path('//?/Global/Z:/'))
    '\\\\?\\Global\\Z:'

      It keeps the trailing slash for some \\.\ paths, but not all.

      Correct:

          >>> str(Path('//./BootPartition/'))
    '\\\\.\\BootPartition\\'

      Incorrect:

          >>> str(Path('//./Global/Z:/'))
    '\\\\.\\Global\\Z:'

      It adds a root directory to \\.\ device paths where none was specified.

      Incorrect:

          >>> str(Path('//./nul'))
    '\\\\.\\nul\\'

    >>> str(Path('//./PhysicalDrive0'))
    '\\\\.\\PhysicalDrive0\\'

    >>> str(Path('//./C:'))
    '\\\\.\\C:\\'

      "\\\\.\\C:" refers to the volume device, whereas "\\\\.\\C:\\" is the root directory in the file system.

      pathlib should parse \\?\ and \\.\ device paths the same way with respect to the drive and root. The difference in practice is only how Windows does (\\.\) or does not (\\?\) canonicalize the path.

      Additionally, pathlib fails to identify the drive correctly in these cases.

      Incorrect:

          >>> Path('//?/Global/Z:/').drive
    '\\\\?\\'

    >>> Path('//?/BootPartition/Temp').drive
    '\\\\?\\'

      Except for "UNC" and "Global" paths, the drive should be the first component after the local-device prefix. The "UNC" device also includes subsequent server and share components, if any. For the reserved "Global" symlink, it should look to the next component. For example, r'\\?\Global\UNC\server\share' is a drive.

      There's also the "GlobalRoot" symlink (or "Global\\GlobalRoot" to be pedantic), but that can't be handled generically.

    2. eryksun
      added
      3.7 (EOL)end of life
      3.8 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      OS-windows
      type-bugAn unexpected behavior, bug, or error
      on Jun 18, 2018

    3. zooba commented on Mar 7, 2019

      @zooba
      zooba
      on Mar 7, 2019
      Member

      Eryk - I've got the PR ready as far as I'm concerned. Have you had a look to see if you're happy with its logic for these cases?

    4. pitrou
      added
      3.9 (EOL)end of life
      on Apr 17, 2020
    5. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    6. brettcannon
      added
      topic-pathlib
      on Jun 21, 2022
    7. eryksun
      changed the title [-]pathlib issues with Windows device paths[/-] [+][pathlib] issues with Windows device paths[/+] on Aug 8, 2022

    8. barneygale commented on Jan 29, 2023

      @barneygale
      barneygale
      on Jan 29, 2023
      Contributor

      @eryksun I suppose that any remaining pathlib device path normalization issues are down to the if condition here:

      cpython/Lib/pathlib.py

      Lines 284 to 286 in 666c084

      if drv.startswith(sep):
      # pathlib assumes that UNC paths always have a root.
      root = sep

      Can you recommend a better condition? :)

    9. zooba commented on Feb 2, 2023

      @zooba
      zooba
      on Feb 2, 2023
      Member

      I think this is the issue that requires splitroot to be correct (which it might be by now?) and then pathlib has to be very deliberate about the meaning of a trailing slash in the drive part vs a leading slash in the rest of the path.

      The current PR doesn't seem to be easily resolvable, but is probably well out of date anyway because of other work that's been done in pathlib and ntpath.

      The existing test suite and the examples Eryk posted above are probably the best way to figure out the right semantics.

    10. barneygale commented on Feb 2, 2023

      @barneygale
      barneygale
      on Feb 2, 2023
      Contributor

      Based on Eryk's test cases, ntpath.splitroot() gets //?/Global and //./Global paths wrong:

      >>> ntpath.splitroot('//./Global/Z:/')
('//./Global', '/', 'Z:/')
>>> ntpath.splitroot('//?/Global/Z:/')
('//?/Global', '/', 'Z:/')

      I believe that all the other failing test cases can be put down to pathlib erroneously adding roots to paths like //./C:, //./nul, //./PhysicalDrive0 and //?/UNC/.

    11. zooba commented on Feb 2, 2023

      @zooba
      zooba
      on Feb 2, 2023
      Member

      I believe that all the other failing test cases can be put down to pathlib erroneously adding roots to paths

      Yeah, this is what I expected with "deliberate about the meaning of ... leading slash in the rest of the path". I know we've encountered these consistency issues before, but I don't know whether they were all resolved.

    12. 2 remaining items

    13. barneygale
      added a commit that references this issue on Feb 17, 2023

      pythonGH-78079: Fix UNC device path root normalization in pathlib

      99f62bc

    14. barneygale commented on Feb 17, 2023

      @barneygale
      barneygale
      on Feb 17, 2023
      Contributor

      I've opened a PR that fixes all of Eryk's test cases except //./Global and //?/Global: #102003

    15. barneygale
      added a commit that references this issue on Feb 20, 2023

      Merge branch 'main' into pythongh-78079-win-device-path-roots

      87519ba
    16. barneygale
      added a commit that references this issue on Mar 10, 2023

      Merge branch 'main' into pythongh-78079-win-device-path-roots

      bb0af91
    17. thunze
      added a commit that references this issue on Mar 21, 2023

      Make `block_device` fixture return `str` instead of `pathlib.Path`

      2309cdd
    18. barneygale
      added 4 commits that reference this issue on Apr 3, 2023

      Merge branch 'main' into pythongh-78079-win-device-path-roots

      f412f87

      Merge branch 'main' into pythongh-78079-win-device-path-roots

      8ff07fc

      Merge branch 'main' into pythongh-78079-win-device-path-roots

      723a5aa

      GH-78079: Fix UNC device path root normalization in pathlib (GH-102003)

      8af8f52

    19. barneygale commented on Apr 14, 2023

      @barneygale
      barneygale
      on Apr 14, 2023
      Contributor

      Hey @eryksun, do you think we should leave this issue open to cover the remaining "Global" and "GLOBALROOT" issues? The problem there lies mostly in ntpath.splitroot(), so I might remove the expert-pathlib label, or even create a new issue.

    20. carljm
      added 2 commits that reference this issue on Apr 17, 2023

      Merge branch 'main' into superopt

      5136459

      Merge branch 'superopt' into superopt_spec

      079c65a

    21. zooba commented on Apr 17, 2023

      @zooba
      zooba
      on Apr 17, 2023
      Member

      I vote for a new issue (with a more descriptive title), and would also say it's not a high priority. If you can describe it well enough, might even be able to justify tagging it "easy" and offer it up for someone at sprints.

    22. barneygale commented on Apr 17, 2023

      @barneygale
      barneygale
      on Apr 17, 2023
      Contributor

      Logged here:

      I don't understand the rules for "Global" and "GLOBALROOT" so it's not very well described :)

      Resolving this issue! Thanks all.

    23. barneygale
      closed this as completedon Apr 17, 2023
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifeOS-windowsstdlibStandard Library Python modules in the Lib/ directorytopic-pathlibtype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions