cformat.rs: fix remaining test_format.test_common_format failures#3405
Conversation
| # TODO: RUSTPYTHON (overflows kill the process immediately in Rust) | ||
| #testcommon("%.*d", (sys.maxsize,1), overflowok=True) # expect overflow |
There was a problem hiding this comment.
I know we don't like doing this, but AIUI running this line doesn't result in an exception at all, or even a panic, but kills the process immediately: I'm not sure what else to do, and this feels like a lesser evil than copying the rest of this test elsewhere (where it won't get updated for new CPython changes to it in future).
There was a problem hiding this comment.
I think this might be a relatively easy error to get. Seems like CPython tries to transform this to an int at some point and this won't work for sys.maxsize. We probably store precision as usize or isize, while it should probably be an i32. Not sure where exactly this is done but it shouldn't be a hard fix (I believe)
There was a problem hiding this comment.
OK, I'll dig into what's causing this. This line should return a string equivalent to "0" * sys.maxsize + "1", so my assumption has been that it's the allocation of that string which is failing, but looking at this again I think that would be a MemoryError.
There was a problem hiding this comment.
I doubt we even handle that yet but I'm not sure tbh. Not sure if RustPython is able to handle the max CPython currently does without crashing.
Looking into the source, 2**31 - 4, assuming 32bit ints, is the max precision could be (see _PyUnicode_FormatLong in unicodeobject.c). Not sure why yet, but it does result in slightly different reporting of the error:
>>> "%.*d" % (2**31, 1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: Python int too large to convert to C int
>>> "%.*d" % (2**31-1, 1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: precision too largeThere was a problem hiding this comment.
OK, so it looks like this is actually a boundary issue:
Welcome to the magnificent Rust Python 0.1.2 interpreter 😱 🖖
>>>>> import sys
>>>>> "%.*d" % (sys.maxsize + 1, 2)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: Python int too large to convert to Rust isize
>>>>> "%.*d" % (sys.maxsize, 2)
memory allocation of 9223372036854775806 bytes failed
[1] 3949127 abort (core dumped) cargo runCPython rejects exactly sys.max_size whereas RustPython accepts sys.max_size (but does raise OverflowErrors, starting at sys.max_size + 1).
I don't think the exact behaviour matters too much here, sys.max_size - 20000 does this:
>>>>> "%.*d" % (sys.maxsize - 20000, 2)
memory allocation of 9223372036854755806 bytes failed
[1] 3949314 abort (core dumped) cargo runand raises MemoryError on CPython: so regardless of the boundary behaviour, it's going to fail.
I think I'll tweak this to test with sys.max_size + 1, so we do trigger the overflow. LMK if you'd prefer a different resolution!
There was a problem hiding this comment.
Hmm, I'm not sure I agree.
With isize, we are more permissive than CPython: we will accept precisions larger than CPython does, but Python code using large widths will behave the same on both CPython and RustPython. With i32, there's Python code which CPython runs which we would reject.
My feeling is that accepting larger precisions than CPython isn't much of a problem: given that the canonical Python implementation rejects them, there's unlikely to be any such Python code out there (and some quick testing locally suggests that such values don't break RustPython).
Am I missing an issue that accepting larger precisions introduces?
There was a problem hiding this comment.
We agree with the test. This might seem pedantic but when tests get updated (and most tests currently do need that!) it's far more likely the whole test will be skipped. The updating process is usually just: copy things over and re-mark as failing/skip the appropriate tests. A small change like this is very likely to be missed by both future contributor and future reviewer (Especially in bulk-updates).
There was a problem hiding this comment.
🤦 I had once again thought that the test was for width, not precison. Fixing now!
There was a problem hiding this comment.
Annoyingly enough, another test case for precision further down in test_format (which is currently skipped altogether) actually uses sys.maxsize + 1 (and also doesn't lead to overflow, just a value error, another slight discrepancy, now between format and % formatting in CPython).
19aabe1 to
af1887e
Compare
This allows it to be used with both self.min_field_width and self.precision, which is necessary for padding out %ds with precision.
This matches CPython's behaviour.
That will always be prepending 0s to %d arguments, the LEFT_ADJUST flag will be used by a later call to `fill_string` with the 0-filled string as the `string` param.
In CPython, width can be isize but precision can only be i32. Our implementation currently assumes the same type for both: as CPython's tests assert on overflows for precision, but not for width, we use that size for both.
Except for the line which raises an OverflowError in CPython, because overflows in Rust (and therefore RustPython) abort the process immediately.
Its string formatting usage now works as expected.
9f4b111 to
7d4f424
Compare
|
Thanks for the patient review, @DimitrisJim! |
With one exception, which I'll comment on inline. Otherwise, commits are atomic.