Sitelet https://github.com/python/cpython/issues/69929
Skip to content

[doc] Clarify exactly what \w matches in UNICODE mode #69929

Closed
Closed
[doc] Clarify exactly what \w matches in UNICODE mode#69929
Labels
3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifedocsDocumentation in the Doc direasytopic-regextype-featureA feature request or enhancement

Description

@zwol
zwol
mannequin
opened on Nov 27, 2015
BPO 25743
Nosy @ezio-melotti, @iritkatriel, @slateny

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-11-27.15:50:58.768>
labels = ['expert-regex', 'easy', '3.9', '3.10', '3.11', 'type-feature', 'docs']
title = '[doc] Clarify exactly what \\w matches in UNICODE mode'
updated_at = <Date 2022-02-28.07:39:46.535>
user = 'https://bugs.python.org/zwol'

bugs.python.org fields:

activity = <Date 2022-02-28.07:39:46.535>
actor = 'slateny'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Regular Expressions']
creation = <Date 2015-11-27.15:50:58.768>
creator = 'zwol'
dependencies = []
files = []
hgrepos = []
issue_num = 25743
keywords = ['easy']
message_count = 5.0
messages = ['255463', '255464', '255465', '407440', '414180']
nosy_count = 7.0
nosy_names = ['ezio.melotti', 'mrabarnett', 'docs@python', 'zwol', 'Andi McClure', 'iritkatriel', 'slateny']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue25743'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

Activity

  1. zwol commented on Nov 27, 2015

    @zwol
    zwolmannequin
    on Nov 27, 2015
    MannequinAuthor

    The re module documentation does not do a good job of explaining exactly what \w matches. Quoting https://docs.python.org/3.5/library/re.html :

    \w
    For Unicode (str) patterns:
    Matches Unicode word characters; this includes most characters
    that can be part of a word in any language, as well as numbers
    and the underscore.

    Empirically, this appears to mean "everything in Unicode general categories L* and N*, plus U+005F (underscore)". That is a perfectly sensible definition and the documentation should state it in those terms. "Unicode word characters" could mean any number of different things; note for instance that UTS#18 gives a very different definition.

    (Further reading: https://gist.github.com/zackw/3077f387591376c7bf67 plus links therefrom).

  2. zwol
    assigned
    docspython
    on Nov 27, 2015
  3. zwol
    added
    docsDocumentation in the Doc dir
    on Nov 27, 2015

  4. AndiMcClure commented on Nov 27, 2015

    @AndiMcClure
    AndiMcCluremannequin
    on Nov 27, 2015
    Mannequin

    I would like to request also a clear explanation be given for the documentation in the 2.7 branch. From https://docs.python.org/2.7/library/re.html :

    "\w ... If UNICODE is set, this will match the characters [0-9_] plus whatever is classified as alphanumeric in the Unicode character properties database"

    This is ambiguous. Does it mean the "Alphabetic" property from UAX#44? Does it mean something else?

  5. zwol commented on Nov 27, 2015

    @zwol
    zwolmannequin
    on Nov 27, 2015
    MannequinAuthor

    FWIW, the actual behavior of \w matching "everything in Unicode general categories L* and N*, plus U+005F (underscore)" is consistent across all versions I can conveniently test (2.7, 3.4, 3.5).

    In 2.7, there are four characters in general category Nl that \w doesn't match, but I believe that is just a bug, not an intentional difference of behavior.

  6. ezio-melotti
    added
    topic-regex
    type-featureA feature request or enhancement
    on Jan 4, 2016

  7. iritkatriel commented on Dec 1, 2021

    @iritkatriel
    iritkatriel
    on Dec 1, 2021
    Member

    It's too late for the 2.7 docs, but the current docs can still be updated.

  8. iritkatriel
    added
    easy
    3.9 (EOL)end of life
    3.10 (EOL)end of life
    3.11only security fixes
    on Dec 1, 2021
  9. iritkatriel
    changed the title [-]Clarify exactly what \w matches in UNICODE mode[/-] [+][doc] Clarify exactly what \w matches in UNICODE mode[/+] on Dec 1, 2021

  10. slateny commented on Feb 28, 2022

    @slateny
    slateny
    on Feb 28, 2022
    Contributor

    Would a change like this be accurate?

    Matches Unicode word characters; this includes most alphanumeric characters as well as the underscore. In Unicode, alphanumeric characters are defined to be the general categories L + N (see https://unicode.org/reports/tr44/#General_Category_Values). If the :const:`ASCII` flag is used, only ``[a-zA-Z0-9_]`` is matched.

  11. ezio-melotti
    transferred this issue fromon Apr 10, 2022
  12. JelleZijlstra
    added a commit that references this issue on Dec 20, 2022

    gh-69929: re docs: Add more specific definition of \w (#92015)

    36a0b1d
  13. miss-islington
    added 4 commits that reference this issue on Dec 20, 2022

    pythongh-69929: re docs: Add more specific definition of \w (pythonGH…

    02c8cc3

    pythongh-69929: re docs: Add more specific definition of \w (pythonGH…

    ee3aeb9

    gh-69929: re docs: Add more specific definition of \w (GH-92015)

    b81d1c3

    gh-69929: re docs: Add more specific definition of \w (GH-92015)

    cfa78ec
  14. slateny
    closed this as completedon Dec 20, 2022
  15. jonburdo
    added a commit that references this issue on Dec 20, 2022

    pythongh-69929: re docs: Add more specific definition of \w (python#9…

    f41cef6
  16. iritkatriel
    added a commit that references this issue on Dec 28, 2022

    Merged main into regmachine branch (#50)

    fdc73cb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifedocsDocumentation in the Doc direasytopic-regextype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions