Sitelet https://github.com/python/cpython/issues/74696
Skip to content

shutil.make_archive should not need to chdir (alternatively: make shutil.make_archive thread-safe) #74696

Closed
Closed
shutil.make_archive should not need to chdir (alternatively: make shutil.make_archive thread-safe)#74696
Assignees
serhiy-storchaka
Labels
stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or errortype-featureA feature request or enhancement

Description

@AlexGaynor
AlexGaynor
mannequin
opened on May 30, 2017
BPO 30511
Nosy @tarekziade, @ambv, @serhiy-storchaka, @miss-islington, @michael-o, @FFY00, @akulakov
PRs
  • bpo-30511: Add note on thread safety to shutil.make_archive() #26933
  • [3.10] bpo-30511: Add note on thread safety to shutil.make_archive() (GH-26933) #27274
  • [3.9] bpo-30511: Add note on thread safety to shutil.make_archive() (GH-26933) #27275
  • [3.8] bpo-30511: Add note on thread safety to shutil.make_archive() (GH-26933) #27276
  • bpo-25625: add contextlib.chdir #28271
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2017-05-30.13:46:18.840>
labels = ['3.7', 'library']
title = 'shutil.make_archive should not need to chdir (alternatively: make shutil.make_archive thread-safe)'
updated_at = <Date 2021-09-17.13:26:13.713>
user = 'https://bugs.python.org/AlexGaynor'

    bugs.python.org fields:

    activity = <Date 2021-09-17.13:26:13.713>
actor = 'FFY00'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-05-30.13:46:18.840>
creator = 'Alex Gaynor'
dependencies = []
files = []
hgrepos = []
issue_num = 30511
keywords = ['patch']
message_count = 10.0
messages = ['294756', '294758', '294759', '294760', '297902', '389849', '396633', '397928', '397932', '397934', '397935']
nosy_count = 9.0
nosy_names = ['tarek', 'lukasz.langa', 'serhiy.storchaka', 'Alex Gaynor', 'Joey Harrington', 'miss-islington', 'michael-o', 'FFY00', 'andrei.avk']
pr_nums = ['26933', '27274', '27275', '27276', '28271']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue30511'
versions = ['Python 2.7', 'Python 3.7']

    Activity

    1. AlexGaynor commented on May 30, 2017

      @AlexGaynor
      AlexGaynormannequin
      on May 30, 2017
      MannequinAuthor

      Currently shutil.make_archive uses os.chdir, however there's no need for that. Everything that's done could be equally accomplished with path manipulation:

      https://github.com/python/cpython/blob/master/Lib/shutil.py#L773-L779

      We should switch to using path manipulation in order to make shutil.make_archive thread safe.

      (Flag: This is probably a good bug for someone with Python skills interested in contributing to CPython!)

    2. AlexGaynor
      added
      3.7 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      on May 30, 2017

    3. serhiy-storchaka commented on May 30, 2017

      @serhiy-storchaka
      serhiy-storchaka
      on May 30, 2017
      Member

      Unfortunately this is not possible. The signature of functions registered with register_archive_format() doesn't support this.

    4. AlexGaynor commented on May 30, 2017

      @AlexGaynor
      AlexGaynormannequin
      on May 30, 2017
      MannequinAuthor

      None of those functions are a public API, so changing them shouldn't be a problem IMO.

    5. AlexGaynor commented on May 30, 2017

      @AlexGaynor
      AlexGaynormannequin
      on May 30, 2017
      MannequinAuthor

      Ugh, except via |register_archive_format|. |register_archive_format| could wrap callables passed to it to maintain the current behavior.

    6. JoeyHarrington commented on Jul 7, 2017

      @JoeyHarrington
      JoeyHarringtonmannequin
      on Jul 7, 2017
      Mannequin

      It would be nice if there was at least a warning in the docs that make_archive is not thread-safe, and that if you have two threads creating archives that it's extremely likely you'll get erroneous results since the race condition lasts for the entire duration of the archive creation.

    7. michael-o commented on Mar 30, 2021

      @michael-o
      michael-omannequin
      on Mar 30, 2021
      Mannequin

      Just wasted two hours for this. Can someone really update the documentation of it if this is not going to change.

    8. akulakov commented on Jun 28, 2021

      @akulakov
      akulakov
      on Jun 28, 2021
      Contributor

      PR is added here:
      #26933

    9. ambv commented on Jul 21, 2021

      @ambv
      ambv
      on Jul 21, 2021
      Contributor

      New changeset 64f54b7 by andrei kulakov in branch 'main':
      bpo-30511: Add note on thread safety to shutil.make_archive() (bpo-26933)
      64f54b7

    10. ambv commented on Jul 21, 2021

      @ambv
      ambv
      on Jul 21, 2021
      Contributor

      New changeset d17449f by Miss Islington (bot) in branch '3.10':
      bpo-30511: Add note on thread safety to shutil.make_archive() (GH-26933) (GH-27274)
      d17449f

    11. ambv commented on Jul 21, 2021

      @ambv
      ambv
      on Jul 21, 2021
      Contributor

      New changeset c8e35ab by Miss Islington (bot) in branch '3.9':
      bpo-30511: Add note on thread safety to shutil.make_archive() (GH-26933) (GH-27275)
      c8e35ab

    12. ambv commented on Jul 21, 2021

      @ambv
      ambv
      on Jul 21, 2021
      Contributor

      New changeset 54d3871 by Miss Islington (bot) in branch '3.8':
      bpo-30511: Add note on thread safety to shutil.make_archive() (GH-26933) (bpo-27276)
      54d3871

    13. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    14. serhiy-storchaka
      added this to Threading issues 🧵on May 20, 2022

    15. 14 remaining items

    16. ambv
      added 2 commits that reference this issue on Jun 22, 2022

      gh-74696: Do not change the current working directory in shutil.make_…

      c1bfff4

      gh-74696: Do not change the current working directory in shutil.make_…

      e51a22b

    17. ambv commented on Jun 22, 2022

      @ambv
      ambv
      on Jun 22, 2022
      Contributor

      @gpshead I'm +1 to deprecate register_archive_format() in 3.12 and remove in 𝛑-Py. Can we use this issue for that or should I open a new one?

    18. serhiy-storchaka commented on Jun 22, 2022

      @serhiy-storchaka
      serhiy-storchaka
      on Jun 22, 2022
      Member

      I was going to extend register_archive_format() by adding support of custom unpackers which support root_dir.

      It can be implemented either by adding a boolean parameter supports_root_dir in register_archive_format(), or checking the supports_root_dir attribute of the unpacker function.

    19. ambv commented on Jun 22, 2022

      @ambv
      ambv
      on Jun 22, 2022
      Contributor

      Sure, Serhiy. Greg's point is that this functionality looks unused so it's probably an unnecessary support burden.

    20. serhiy-storchaka commented on Jun 22, 2022

      @serhiy-storchaka
      serhiy-storchaka
      on Jun 22, 2022
      Member

      py7zr provides helpers to be registered in shutil. Other archiver modules can provide such helpers too or even register them by default. Removing register_archive_format() and register_unpack_format() will make other archive formats second-class citizens.

    21. serhiy-storchaka
      added a commit that references this issue on Jun 25, 2022

      pythongh-74696: Pass root_dir to custom archivers which support it

      e5ebb8d

    22. serhiy-storchaka commented on Jun 25, 2022

      @serhiy-storchaka
      serhiy-storchaka
      on Jun 25, 2022
      Member

      I have implemented the second option in #94251. It seems better because does not need a collaboration between the author and users of the custom archiver.

    23. serhiy-storchaka
      added a commit that references this issue on Jun 26, 2022

      Revert 30935 defer bpo45162 to 312 (#94285)

      1ffde3e
    24. serhiy-storchaka
      added a commit that references this issue on Oct 5, 2022

      gh-74696: Pass root_dir to custom archivers which support it (GH-94251)

      e3ef400
    25. carljm
      added a commit that references this issue on Oct 6, 2022

      Merge branch 'main' into dictwatch

      4bb5d9d
    26. mpage
      added a commit that references this issue on Oct 11, 2022

      pythongh-74696: Pass root_dir to custom archivers which support it (p…

      c34bef6
    27. serhiy-storchaka
      closed this as completedon Oct 27, 2022
    28. Repository owner moved this from Todo to Done in Threading issues 🧵on Oct 27, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or errortype-featureA feature request or enhancement

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions