Sitelet https://github.com/python/cpython/issues/91575
Skip to content

Outdated Unicode data in the re module #91575

Description

@serhiy-storchaka
  1. The re module contains a table for characters: c1.upper() == c2.upper() and c1 != c2 and c1.lower() == c1 and c2.lower() == c2. For example, 'ς' and 'σ': 'ς'.upper() == 'σ'.upper() == 'Σ'.

    It was generated for 3.5. But newer Python versions support newer Unicode standards, and more such characters were added. For example: 'в' and 'ᲀ': 'в'.upper() == 'ᲀ'.upper() == 'В'.

    Python re lib fails case insensitive matches on Unicode data #56937

  2. The code depends on some assumption about characters outside of the BMP range. The comment says that there are only two ranges of cased non-BMP characters, and that RANGE_UNI_IGNORE works with them.

    Now there are more ranges of cased non-BMP characters. Seems the assumption is still true and RANGE_UNI_IGNORE still works, but the comment is outdated.

    IGNORECASE breaks unicode literal range matching #61583

The plan is:

  • Regenerate the table with actual Unicode versions for all maintained Python versions.
  • Test the assumption and update the comment.
  • Add a script and the make target for generating that table with the actual Unicode version (the developed version only).
  • For the above assumption, either test it in the script, or make the code working in case it is not true.

Activity

  1. added a commit that references this issue on Apr 15, 2022
  2. serhiy-storchaka commented on Apr 16, 2022

    @serhiy-storchaka
    MemberAuthor

    I am already working on this.

  3. added a commit that references this issue on Apr 18, 2022
  4. added a commit that references this issue on Apr 22, 2022
  5. added a commit that references this issue on Apr 22, 2022
  6. added 2 commits that reference this issue on Apr 22, 2022
  7. added a commit that references this issue on Jun 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifetopic-regextype-bugAn unexpected behavior, bug, or errortype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions