Sitelet https://web.archive.org/web/20260616114616/https://github.com/RustPython/RustPython/pull/3405
Skip to content

cformat.rs: fix remaining test_format.test_common_format failures#3405

Merged
DimitrisJim merged 7 commits into
RustPython:mainfrom
OddBloke:oddbloke/format
Nov 5, 2021
Merged

cformat.rs: fix remaining test_format.test_common_format failures#3405
DimitrisJim merged 7 commits into
RustPython:mainfrom
OddBloke:oddbloke/format

Conversation

@OddBloke

@OddBloke OddBloke commented Nov 2, 2021

Copy link
Copy Markdown
Collaborator

With one exception, which I'll comment on inline. Otherwise, commits are atomic.

Comment thread Lib/test/test_format.py Outdated
Comment on lines +102 to +103
# TODO: RUSTPYTHON (overflows kill the process immediately in Rust)
#testcommon("%.*d", (sys.maxsize,1), overflowok=True) # expect overflow

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know we don't like doing this, but AIUI running this line doesn't result in an exception at all, or even a panic, but kills the process immediately: I'm not sure what else to do, and this feels like a lesser evil than copying the rest of this test elsewhere (where it won't get updated for new CPython changes to it in future).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this might be a relatively easy error to get. Seems like CPython tries to transform this to an int at some point and this won't work for sys.maxsize. We probably store precision as usize or isize, while it should probably be an i32. Not sure where exactly this is done but it shouldn't be a hard fix (I believe)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I'll dig into what's causing this. This line should return a string equivalent to "0" * sys.maxsize + "1", so my assumption has been that it's the allocation of that string which is failing, but looking at this again I think that would be a MemoryError.

@DimitrisJim DimitrisJim Nov 3, 2021 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I doubt we even handle that yet but I'm not sure tbh. Not sure if RustPython is able to handle the max CPython currently does without crashing.

Looking into the source, 2**31 - 4, assuming 32bit ints, is the max precision could be (see _PyUnicode_FormatLong in unicodeobject.c). Not sure why yet, but it does result in slightly different reporting of the error:

 >>> "%.*d" % (2**31, 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: Python int too large to convert to C int
>>> "%.*d" % (2**31-1, 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: precision too large

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, so it looks like this is actually a boundary issue:

Welcome to the magnificent Rust Python 0.1.2 interpreter 😱 🖖
>>>>> import sys
>>>>> "%.*d" % (sys.maxsize + 1, 2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: Python int too large to convert to Rust isize
>>>>> "%.*d" % (sys.maxsize, 2)
memory allocation of 9223372036854775806 bytes failed
[1]    3949127 abort (core dumped)  cargo run

CPython rejects exactly sys.max_size whereas RustPython accepts sys.max_size (but does raise OverflowErrors, starting at sys.max_size + 1).

I don't think the exact behaviour matters too much here, sys.max_size - 20000 does this:

>>>>> "%.*d" % (sys.maxsize - 20000, 2)
memory allocation of 9223372036854755806 bytes failed
[1]    3949314 abort (core dumped)  cargo run

and raises MemoryError on CPython: so regardless of the boundary behaviour, it's going to fail.

I think I'll tweak this to test with sys.max_size + 1, so we do trigger the overflow. LMK if you'd prefer a different resolution!

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I'm not sure I agree.

With isize, we are more permissive than CPython: we will accept precisions larger than CPython does, but Python code using large widths will behave the same on both CPython and RustPython. With i32, there's Python code which CPython runs which we would reject.

My feeling is that accepting larger precisions than CPython isn't much of a problem: given that the canonical Python implementation rejects them, there's unlikely to be any such Python code out there (and some quick testing locally suggests that such values don't break RustPython).

Am I missing an issue that accepting larger precisions introduces?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We agree with the test. This might seem pedantic but when tests get updated (and most tests currently do need that!) it's far more likely the whole test will be skipped. The updating process is usually just: copy things over and re-mark as failing/skip the appropriate tests. A small change like this is very likely to be missed by both future contributor and future reviewer (Especially in bulk-updates).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤦 I had once again thought that the test was for width, not precison. Fixing now!

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@DimitrisJim DimitrisJim Nov 4, 2021 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Annoyingly enough, another test case for precision further down in test_format (which is currently skipped altogether) actually uses sys.maxsize + 1 (and also doesn't lead to overflow, just a value error, another slight discrepancy, now between format and % formatting in CPython).

@OddBloke OddBloke force-pushed the oddbloke/format branch 2 times, most recently from 19aabe1 to af1887e Compare November 3, 2021 22:12
This allows it to be used with both self.min_field_width and
self.precision, which is necessary for padding out %ds with precision.
This matches CPython's behaviour.
That will always be prepending 0s to %d arguments, the LEFT_ADJUST flag
will be used by a later call to `fill_string` with the 0-filled string
as the `string` param.
In CPython, width can be isize but precision can only be i32.  Our
implementation currently assumes the same type for both: as CPython's
tests assert on overflows for precision, but not for width, we use that
size for both.
Except for the line which raises an OverflowError in CPython, because
overflows in Rust (and therefore RustPython) abort the process
immediately.
Its string formatting usage now works as expected.
@DimitrisJim DimitrisJim merged commit 14b4f00 into RustPython:main Nov 5, 2021
@OddBloke OddBloke deleted the oddbloke/format branch November 5, 2021 12:52
@OddBloke

OddBloke commented Nov 5, 2021

Copy link
Copy Markdown
Collaborator Author

Thanks for the patient review, @DimitrisJim!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants