Sitelet https://github.com/python/cpython/issues/91415
Skip to content

Clarify SortingHOWTO regarding locale aware string sorting #91415

Closed
Closed
Clarify SortingHOWTO regarding locale aware string sorting#91415
Assignees
rhettinger
Labels
3.10 (EOL)end of life3.11only security fixesdocsDocumentation in the Doc dir

Description

@CendioOssman
CendioOssman
mannequin
opened on Apr 8, 2022
BPO 47259
Nosy @rhettinger, @stevendaprano, @CendioOssman

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = 'https://github.com/rhettinger'
closed_at = None
created_at = <Date 2022-04-08.10:31:59.766>
labels = ['3.11', '3.10', 'docs']
title = 'Clarify SortingHOWTO regarding locale aware string sorting'
updated_at = <Date 2022-04-08.17:49:04.631>
user = 'https://github.com/CendioOssman'

bugs.python.org fields:

activity = <Date 2022-04-08.17:49:04.631>
actor = 'rhettinger'
assignee = 'rhettinger'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2022-04-08.10:31:59.766>
creator = 'CendioOssman'
dependencies = []
files = []
hgrepos = []
issue_num = 47259
keywords = []
message_count = 2.0
messages = ['416972', '416997']
nosy_count = 3.0
nosy_names = ['rhettinger', 'steven.daprano', 'CendioOssman']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue47259'
versions = ['Python 3.10', 'Python 3.11']

Activity

  1. CendioOssman commented on Apr 8, 2022

    @CendioOssman
    CendioOssmanmannequin
    on Apr 8, 2022
    MannequinAuthor

    There is a big gotcha in Python that is easily overlooked and should at the very least be more prominently pointed out in the documentation.

    Sorting strings will produce results that is very confusing for humans.

    I happens to work for ASCII, but will generally produce bad results for other things as code points do not always follow the alphabetical order.

    The expressions chapter¹ mentions this fact, but you have to dig quite a bit to reach that. It also mentions that normalization is an issue, but it never mentions the issue about code point order versus alphabetical order.

    The sorting tutorial mentions under "Odds and ends"² that you need to use a special key or comparison function to get locale aware sorting. It doesn't mention that this also includes respecting alphabetical order, which might be overlooked unless you are very familiar with how the sorting works. The tutorial is also something you have to dig a bit to reach.

    Ideally string comparison would always be locale aware in a high level language such as Python. However, a smaller step would be a note on sorted()³ that extra care needs to be taken for strings as the default behaviour will produce unexpected results once your strings include anything outside the English alphabet.

    ¹ https://docs.python.org/3/reference/expressions.html
    ² https://docs.python.org/3/howto/sorting.html#odd-and-ends
    ³ https://docs.python.org/3/library/functions.html#sorted

  2. CendioOssman
    added
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    on Apr 8, 2022

  3. rhettinger commented on Apr 8, 2022

    @rhettinger
    rhettinger
    on Apr 8, 2022
    Contributor

    I don't think splashing this everywhere else in the docs would be helpful. Tools like list.sort, sorted, min, max, nlargest, nsmallest use whatever sort order is provided by the underlying object whether it be a string, tuple, float, or int.

    The section on expressions is the intended place to cover how comparison are defined for core objects: https://docs.python.org/3/reference/expressions.html#value-comparisons

    As suggested, I will edit the sorting howto to be cleared that locale aware sort ordering refers to alphabetical orderings which can vary (for example, the Spanish ll sorts differently in different locales).

  4. rhettinger
    added
    docsDocumentation in the Doc dir
    and removed
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    on Apr 8, 2022
  5. rhettinger
    self-assigned this
    on Apr 8, 2022
  6. rhettinger
    added
    docsDocumentation in the Doc dir
    and removed
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    on Apr 8, 2022
  7. rhettinger
    self-assigned this
    on Apr 8, 2022
  8. rhettinger
    added
    3.10 (EOL)end of life
    3.11only security fixes
    on Apr 8, 2022
  9. rhettinger
    changed the title [-]string sorting often incorrect[/-] [+]Clarify SortingHOWTO regarding locale aware string sorting[/+] on Apr 8, 2022
  10. rhettinger
    added
    3.10 (EOL)end of life
    3.11only security fixes
    on Apr 8, 2022
  11. rhettinger
    changed the title [-]string sorting often incorrect[/-] [+]Clarify SortingHOWTO regarding locale aware string sorting[/+] on Apr 8, 2022
  12. ezio-melotti
    transferred this issue fromon Apr 10, 2022

  13. andjc commented on Apr 14, 2022

    @andjc
    andjc
    on Apr 14, 2022

    Actually sorting, and collation in general, can be quite a complex topic, I am slowly working through writing my own notes on it, and it can bend the mind at times.

    You have various approaches to sorting, including:

    • ISO/IEC 14651
    • normalised ISO/IEC 14651
    • language tailored ISO/IEC 14651
    • normalised language tailored ISO/IEC 14651
    • EOR (either ISO/IEC 14651 based implementation or CLDR based implementation)
    • DUCET
    • CLDR collation algorithm
    • language tailored CLDR collation
    • customisations to the CLDR collation algorithm or language tailored CLDR collations

    You could probably write a whole book just to properly address available customisations to CLDR collations.

    Added to that, you have the differences in locales between GNU/Linux, BSD/macOS, and Windows, and the fact that many locale collation tables on BSD and macOS are symlinked to one specific collation table (which negates the ability for language tailored collation).

    Then you have emoji collation sequences and the possibility of building custom collation rules, combining language tailored collation with emoji collation.

  14. serhiy-storchaka
    added this to Locale issues 🗺on May 21, 2022
  15. rhettinger
    added 2 commits that reference this issue on Oct 16, 2022

    pythonGH-91415: Mention alphabetical sort ordering in the Sorting HOWTO

    7830a51

    GH-91415: Mention alphabetical sort ordering in the Sorting HOWTO (GH…

    ae19217
  16. miss-islington
    added a commit that references this issue on Oct 16, 2022

    pythonGH-91415: Mention alphabetical sort ordering in the Sorting HOW…

    4a02bae
  17. rhettinger
    closed this as completedon Oct 16, 2022
  18. rhettinger
    moved this to Done in Locale issues 🗺on Oct 16, 2022
  19. miss-islington
    added a commit that references this issue on Oct 16, 2022

    GH-91415: Mention alphabetical sort ordering in the Sorting HOWTO (GH…

    211b819
  20. carljm
    added a commit that references this issue on Oct 17, 2022

    Merge branch 'main' into typewatch

    cdbe93b
  21. pablogsal
    added a commit that references this issue on Oct 22, 2022

    GH-91415: Mention alphabetical sort ordering in the Sorting HOWTO (GH…

    c6df6ee
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.10 (EOL)end of life3.11only security fixesdocsDocumentation in the Doc dir

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions