Sitelet https://github.com/python/cpython/commit/d19febb00c5a5ff20fee4b251a9656d088cf6965
Skip to content

Commit d19febb

Browse files
vstinnervstinner
authored
Rework locale encoding C functions (#158680)
Add output_length to _Py_EncodeLocale(), _Py_EncodeUTF8(), encode_current_locale() and encode_ascii(). So unicode_encode_locale() and wstr_to_utf8() can now use the output_length, instead of having to compute strlen(). * Rename functions: * _Py_DecodeUTF8Ex() => _Py_DecodeUTF8() * _Py_EncodeUTF8Ex() => _Py_EncodeUTF8() * _Py_EncodeLocaleEx() => _Py_EncodeLocale() * _Py_DecodeLocaleEx() => _Py_DecodeLocale() * Add tests on Py_DecodeLocale() and Py_EncodeLocale() in test_codecs. Clarify also their documentation. * _Py_EncodeLocale() and _Py_DecodeLocale() now sets all output parameters on error. Add more tests on these output parameters. * Add encode_current_locale_impl() to simplify encode_current_locale(). * Remove reason parameter of _Py_EncodeLocale() and _Py_DecodeLocale() to simplify the code. Only the UTF-8 decoder used specific reasons. * _Py_EncodeUTF8() now uses size_t instead of Py_ssize_t to iterate on the input string. * On error, encode_current_locale_impl() now returns ENCODE_ERROR (new constant) instead of DECODE_ERROR. * Mark _Py_EncodeLocaleRaw() as static (it's not used outside fileutils.c).
1 parent a7b5fc5 commit d19febb

8 files changed

Lines changed: 734 additions & 435 deletions

File tree

‎Doc/c-api/sys.rst‎

Lines changed: 23 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -152,28 +152,29 @@ Operating System Utilities
152152
<c-preinit>` and so that the LC_CTYPE locale is properly configured: see
153153
the :c:func:`Py_PreInitialize` function.
154154
155-
Decode a byte string from the :term:`filesystem encoding and error handler`.
156-
If the error handler is :ref:`surrogateescape error handler
157-
<surrogateescape>`, undecodable bytes are decoded as characters in range
158-
U+DC80..U+DCFF; and if a byte sequence can be decoded as a surrogate
159-
character, the bytes are escaped using the surrogateescape error handler
160-
instead of decoding them.
155+
Decode a byte string from the :term:`filesystem encoding <filesystem
156+
encoding and error handler>` with the :ref:`surrogateescape error handler
157+
<surrogateescape>`.
158+
159+
Undecodable bytes are decoded as characters in range U+DC80..U+DCFF. If a
160+
byte sequence can be decoded as a surrogate character, escape the bytes
161+
using the surrogateescape error handler instead of decoding them.
161162
162163
Return a pointer to a newly allocated wide character string, use
163164
:c:func:`PyMem_RawFree` to free the memory. If size is not ``NULL``, write
164165
the number of wide characters excluding the null character into ``*size``
165166
166-
Return ``NULL`` on decoding error or memory allocation error. If *size* is
167-
not ``NULL``, ``*size`` is set to ``(size_t)-1`` on memory error or set to
168-
``(size_t)-2`` on decoding error.
167+
On memory allocation failure, set *\*size* to ``(size_t)-1`` and return
168+
``NULL``.
169+
170+
On decode error, set *\*size* to ``(size_t)-2`` and return ``NULL``.
171+
Decoding errors should never happen, unless there is a bug in the C
172+
library.
169173
170174
The :term:`filesystem encoding and error handler` are selected by
171175
:c:func:`PyConfig_Read`: see :c:member:`~PyConfig.filesystem_encoding` and
172176
:c:member:`~PyConfig.filesystem_errors` members of :c:type:`PyConfig`.
173177
174-
Decoding errors should never happen, unless there is a bug in the C
175-
library.
176-
177178
Use the :c:func:`Py_EncodeLocale` function to encode the character string
178179
back to a byte string.
179180
@@ -195,17 +196,19 @@ Operating System Utilities
195196
196197
.. c:function:: char* Py_EncodeLocale(const wchar_t *text, size_t *error_pos)
197198
198-
Encode a wide character string to the :term:`filesystem encoding and error
199-
handler`. If the error handler is :ref:`surrogateescape error handler
200-
<surrogateescape>`, surrogate characters in the range U+DC80..U+DCFF are
201-
converted to bytes 0x80..0xFF.
199+
Encode a wide character string to the :term:`filesystem encoding <filesystem
200+
encoding and error handler>` with the :ref:`surrogateescape error handler
201+
<surrogateescape>`. Surrogate characters in the range U+DC80..U+DCFF are
202+
encoded to bytes 0x80..0xFF.
202203
203204
Return a pointer to a newly allocated byte string, use :c:func:`PyMem_Free`
204-
to free the memory. Return ``NULL`` on encoding error or memory allocation
205-
error.
205+
to free the memory.
206+
207+
On memory allocation failure, set *\*error_pos* to ``(size_t)-1`` and return
208+
``NULL``.
206209
207-
If error_pos is not ``NULL``, ``*error_pos`` is set to ``(size_t)-1`` on
208-
success, or set to the index of the invalid character on encoding error.
210+
On encoding error, set *\*error_pos* to the index of the first unencodable
211+
character and return ``NULL``.
209212
210213
The :term:`filesystem encoding and error handler` are selected by
211214
:c:func:`PyConfig_Read`: see :c:member:`~PyConfig.filesystem_encoding` and

‎Include/internal/pycore_fileutils.h‎

Lines changed: 11 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -24,27 +24,22 @@ extern "C" {
2424
PyAPI_FUNC(_Py_error_handler) _Py_GetErrorHandler(const char *errors);
2525

2626
// Export for '_testinternalcapi' shared extension
27-
PyAPI_FUNC(int) _Py_DecodeLocaleEx(
27+
PyAPI_FUNC(int) _Py_DecodeLocale(
2828
const char *arg,
2929
wchar_t **wstr,
3030
size_t *wlen,
31-
const char **reason,
3231
int current_locale,
3332
_Py_error_handler errors);
3433

3534
// Export for '_testinternalcapi' shared extension
36-
PyAPI_FUNC(int) _Py_EncodeLocaleEx(
35+
PyAPI_FUNC(int) _Py_EncodeLocale(
3736
const wchar_t *text,
3837
char **str,
38+
size_t *output_length,
3939
size_t *error_pos,
40-
const char **reason,
4140
int current_locale,
4241
_Py_error_handler errors);
4342

44-
extern char* _Py_EncodeLocaleRaw(
45-
const wchar_t *text,
46-
size_t *error_pos);
47-
4843
extern PyObject* _Py_device_encoding(int);
4944

5045
#if defined(MS_WINDOWS) || defined(__APPLE__)
@@ -190,19 +185,23 @@ extern int _Py_open_osfhandle(void *handle, int flags);
190185
? _PyStatus_ERR("cannot decode " NAME) \
191186
: _PyStatus_NO_MEMORY()
192187

193-
extern int _Py_DecodeUTF8Ex(
188+
#define _Py_CODEC_MEMORY_ERROR -1
189+
#define _Py_CODEC_DECODE_ERROR -2
190+
#define _Py_CODEC_ENCODE_ERROR -2
191+
#define _Py_CODEC_UNSUPPORTED_ERROR_HANDLER -3
192+
193+
extern int _Py_DecodeUTF8(
194194
const char *arg,
195195
Py_ssize_t arglen,
196196
wchar_t **wstr,
197197
size_t *wlen,
198-
const char **reason,
199198
_Py_error_handler errors);
200199

201-
extern int _Py_EncodeUTF8Ex(
200+
extern int _Py_EncodeUTF8(
202201
const wchar_t *text,
203202
char **str,
203+
size_t *output_length,
204204
size_t *error_pos,
205-
const char **reason,
206205
int raw_malloc,
207206
_Py_error_handler errors);
208207

‎Lib/test/test_codecs.py‎

Lines changed: 82 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -4057,34 +4057,67 @@ def test_pickle(self):
40574057
pickle.dumps(sr, proto)
40584058

40594059

4060+
@unittest.skipIf(_testlimitedcapi is None, 'need _testlimitedcapi module')
40604061
@unittest.skipIf(_testinternalcapi is None, 'need _testinternalcapi module')
40614062
class LocaleCodecTest(unittest.TestCase):
40624063
"""
4063-
Test indirectly _Py_DecodeUTF8Ex() and _Py_EncodeUTF8Ex().
4064+
Test public Py_EncodeLocale() and Py_DecodeLocale() C API.
4065+
4066+
Test internal _Py_EncodeLocale() and _Py_DecodeLocale() C API.
4067+
4068+
Test indirectly _Py_DecodeUTF8() and _Py_EncodeUTF8().
40644069
"""
40654070
ENCODING = sys.getfilesystemencoding()
40664071
STRINGS = ("ascii", "ulatin1:\xa7\xe9",
40674072
"u255:\xff",
40684073
"UCS:\xe9\u20ac\U0010ffff",
4069-
"surrogates:\uDC80\uDCFF")
4074+
"surrogates:\uDC80\uDCFF",
4075+
"embed\0char")
40704076
BYTES_STRINGS = (b"blatin1:\xa7\xe9", b"b255:\xff")
4071-
SURROGATES = "\uDC80\uDCFF"
40724077

4073-
def encode(self, text, errors="strict"):
4074-
return _testinternalcapi.EncodeLocaleEx(text, 0, errors)
4078+
def encode_locale_surrogateescape(self, text):
4079+
# Test public Py_EncodeLocale() C API:
4080+
# use the "surrogateescape" error handler
4081+
return _testlimitedcapi.encode_locale(text)
4082+
4083+
def encode_locale(self, text, errors="strict"):
4084+
# Test internal _Py_EncodeLocale() C API
4085+
return _testinternalcapi.encode_locale(text, 0, errors)
40754086

40764087
def check_encode_strings(self, errors):
40774088
for text in self.STRINGS:
40784089
with self.subTest(text=text):
40794090
try:
40804091
expected = text.encode(self.ENCODING, errors)
4092+
if b"\0" in expected:
4093+
# Py_EncodeLocale() and _Py_EncodeLocale()
4094+
# truncate the input string at the first NUL character
4095+
expected = expected.partition(b'\0')[0]
40814096
except UnicodeEncodeError:
4097+
for error_pos in range(len(text)):
4098+
try:
4099+
text[error_pos].encode(self.ENCODING, errors)
4100+
except UnicodeEncodeError:
4101+
break
4102+
else:
4103+
self.fail("failed to compute error_pos")
4104+
4105+
if errors == "surrogateescape":
4106+
with self.assertRaises(RuntimeError) as cm:
4107+
self.encode_locale_surrogateescape(text)
4108+
errmsg = f"encode error: pos={error_pos}"
4109+
self.assertEqual(str(cm.exception), errmsg)
4110+
40824111
with self.assertRaises(RuntimeError) as cm:
4083-
self.encode(text, errors)
4084-
errmsg = str(cm.exception)
4085-
self.assertRegex(errmsg, r"encode error: pos=[0-9]+, reason=")
4112+
self.encode_locale(text, errors)
4113+
errmsg = f"encode error: pos={error_pos}"
4114+
self.assertEqual(str(cm.exception), errmsg)
40864115
else:
4087-
encoded = self.encode(text, errors)
4116+
if errors in ("strict", "surrogateescape"):
4117+
encoded = self.encode_locale_surrogateescape(text)
4118+
self.assertEqual(encoded, expected)
4119+
4120+
encoded = self.encode_locale(text, errors)
40884121
self.assertEqual(encoded, expected)
40894122

40904123
def test_encode_strict(self):
@@ -4095,7 +4128,7 @@ def test_encode_surrogateescape(self):
40954128

40964129
def test_encode_surrogatepass(self):
40974130
try:
4098-
self.encode('', 'surrogatepass')
4131+
self.encode_locale('', 'surrogatepass')
40994132
except ValueError as exc:
41004133
if str(exc) == 'unsupported error handler':
41014134
self.skipTest(f"{self.ENCODING!r} encoder doesn't support "
@@ -4107,11 +4140,17 @@ def test_encode_surrogatepass(self):
41074140

41084141
def test_encode_unsupported_error_handler(self):
41094142
with self.assertRaises(ValueError) as cm:
4110-
self.encode('', 'backslashreplace')
4143+
self.encode_locale('', 'backslashreplace')
41114144
self.assertEqual(str(cm.exception), 'unsupported error handler')
41124145

4113-
def decode(self, encoded, errors="strict"):
4114-
return _testinternalcapi.DecodeLocaleEx(encoded, 0, errors)
4146+
def decode_locale(self, encoded, errors="strict"):
4147+
# Test internal _Py_DecodeLocale() C API
4148+
return _testinternalcapi.decode_locale(encoded, 0, errors)
4149+
4150+
def decode_locale_surrogateescape(self, encoded):
4151+
# Test the public Py_DecodeLocale() C API:
4152+
# use the "surrogateescape" error handler
4153+
return _testlimitedcapi.decode_locale(encoded)
41154154

41164155
def check_decode_strings(self, errors):
41174156
is_utf8 = (self.ENCODING == "utf-8")
@@ -4138,13 +4177,37 @@ def check_decode_strings(self, errors):
41384177
with self.subTest(encoded=encoded):
41394178
try:
41404179
expected = encoded.decode(self.ENCODING, errors)
4180+
if "\0" in expected:
4181+
# Py_DecodeLocale() and _Py_DecodeLocale() truncate
4182+
# the input string at the first NUL byte
4183+
expected = expected.partition('\0')[0]
41414184
except UnicodeDecodeError:
4185+
for error_pos in range(len(encoded) - 1, -1, -1):
4186+
try:
4187+
encoded[:error_pos].decode(self.ENCODING, errors)
4188+
except UnicodeDecodeError:
4189+
pass
4190+
else:
4191+
break
4192+
else:
4193+
self.fail("failed to compute error_pos")
4194+
4195+
if errors == "surrogateescape":
4196+
with self.assertRaises(RuntimeError) as cm:
4197+
self.decode_locale_surrogateescape(encoded)
4198+
errmsg = f"Py_DecodeLocale failed: error_pos={error_pos}"
4199+
self.assertEqual(str(cm.exception), errmsg)
4200+
41424201
with self.assertRaises(RuntimeError) as cm:
4143-
self.decode(encoded, errors)
4144-
errmsg = str(cm.exception)
4145-
self.assertStartsWith(errmsg, "decode error: ")
4202+
self.decode_locale(encoded, errors)
4203+
errmsg = f"decode error: pos={error_pos}"
4204+
self.assertEqual(str(cm.exception), errmsg)
41464205
else:
4147-
decoded = self.decode(encoded, errors)
4206+
if errors == ("strict", "surrogateescape"):
4207+
decoded = self.decode_locale_surrogateescape(encoded)
4208+
self.assertEqual(decoded, expected)
4209+
4210+
decoded = self.decode_locale(encoded, errors)
41484211
self.assertEqual(decoded, expected)
41494212

41504213
def test_decode_strict(self):
@@ -4155,7 +4218,7 @@ def test_decode_surrogateescape(self):
41554218

41564219
def test_decode_surrogatepass(self):
41574220
try:
4158-
self.decode(b'', 'surrogatepass')
4221+
self.decode_locale(b'', 'surrogatepass')
41594222
except ValueError as exc:
41604223
if str(exc) == 'unsupported error handler':
41614224
self.skipTest(f"{self.ENCODING!r} decoder doesn't support "
@@ -4167,7 +4230,7 @@ def test_decode_surrogatepass(self):
41674230

41684231
def test_decode_unsupported_error_handler(self):
41694232
with self.assertRaises(ValueError) as cm:
4170-
self.decode(b'', 'backslashreplace')
4233+
self.decode_locale(b'', 'backslashreplace')
41714234
self.assertEqual(str(cm.exception), 'unsupported error handler')
41724235

41734236

0 commit comments

Comments
 (0)