bpo-38202: PyObject_GetIter() should be checked for failure #16241

ZackerySpytz · 2019-09-17T19:29:32Z

An exception may occur during a PyObject_GetIter() call.

https://bugs.python.org/issue38202

An exception may occur during a PyObject_GetIter() call.

serhiy-storchaka · 2019-09-17T21:13:41Z

Could you add a test?

ZackerySpytz · 2019-10-04T09:57:21Z

@serhiy-storchaka The fatal error I reported on the BPO issue was due to a MemoryError in PyObject_GetIter(). I don't think an exception in this PyObject_GetIter() call could be triggered from the Python side.

serhiy-storchaka · 2019-10-04T10:05:34Z

It can be. For example:

$ ./python -c '{}.keys() & 1'
python: Objects/typeobject.c:3148: _PyType_Lookup: Assertion `!PyErr_Occurred()' failed.
Aborted (core dumped)

corona10

Please add misc/news through the blurb.
And add unit tests based on @serhiy-storchaka suggested.
intersection, or, xor, sub operations should be added.

corona10 · 2019-10-09T08:02:12Z

@methane @serhiy-storchaka
I'd like to add the unit test if this PR is merged without unit test.
Is it okay?

pablogsal · 2019-10-10T21:33:15Z

I have added a unit test. Without the current PR:

❯ ./python -m test test_dictviews
Run tests sequentially
0:00:00 load avg: 1.94 [1/1] test_dictviews
python: Objects/typeobject.c:3125: _PyType_Lookup: Assertion `!PyErr_Occurred()' failed.
Fatal Python error: Aborted

Current thread 0x00007f0fa489c740 (most recent call first):
[1]    26485 abort (core dumped)  ./python -m test test_dictviews

now:

 ./python -m test test_dictviews
Run tests sequentially
0:00:00 load avg: 0.69 [1/1] test_dictviews

== Tests result: SUCCESS ==

1 test OK.

Total duration: 45 ms
Tests result: SUCCESS

pablogsal · 2019-10-10T21:39:10Z

Thanks @serhiy-storchaka !

serhiy-storchaka · 2019-10-10T21:39:51Z

Thanks @pablogsal!

bedevere-bot · 2019-10-13T12:55:12Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian PGO 3.x has failed when building commit b16e382.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/47/builds/3699) and take a look at the build logs.
Check if the failure is related to this commit (b16e382) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/47/builds/3699

Failed tests:

test_multiprocessing_fork

Summary of the results of the build (if available):

==

Click to see traceback logs

From https://github.com/python/cpython
 * branch                  master     -> FETCH_HEAD
Reset branch 'master'

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make[2]: [clean-retain-profile] Error 1 (ignored)
/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Modules/expat/xmlparse.c: In function ‘appendAttributeValue’:
/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Modules/expat/xmlparse.c:5329:39: warning: array subscript is above array bounds [-Warray-bounds]
         if (! poolAppendChar(pool, buf[i]))
                                    ~~~^~~
/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Modules/expat/xmlparse.c:514:31: note: in definition of macro ‘poolAppendChar’
        : ((*((pool)->ptr)++ = c), 1))
                               ^

test_devpoll skipped -- test works only on Solaris OS family
stty: 'standard input': Inappropriate ioctl for device
test_ttk_guionly skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_kqueue skipped -- test works only on BSD
test_msilib skipped -- No module named '_msi'
test_winconsoleio skipped -- test only relevant on win32
test_ossaudiodev skipped -- [Errno 2] No such file or directory: '/dev/dsp'
test_tix skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_tk skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_winsound skipped -- No module named 'winsound'
test_gdb skipped -- test_gdb is not reliable on PGO builds
test_winreg skipped -- No module named 'winreg'
test_startfile skipped -- object <module 'os' from '/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/os.py'> has no attribute 'startfile'
test_ioctl skipped -- Unable to open /dev/tty
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
test_flock (__main__.FNTLEINTRTest) ... ok
test_lockf (__main__.FNTLEINTRTest) ... ok
test_read (__main__.OSEINTRTest) ... ok
test_wait (__main__.OSEINTRTest) ... ok
test_wait3 (__main__.OSEINTRTest) ... ok
test_wait4 (__main__.OSEINTRTest) ... ok
test_waitpid (__main__.OSEINTRTest) ... ok
test_write (__main__.OSEINTRTest) ... ok
test_devpoll (__main__.SelectEINTRTest) ... skipped 'need select.devpoll'
test_epoll (__main__.SelectEINTRTest) ... ok
test_kqueue (__main__.SelectEINTRTest) ... skipped 'need select.kqueue'
test_poll (__main__.SelectEINTRTest) ... ok
test_select (__main__.SelectEINTRTest) ... ok
test_sigtimedwait (__main__.SignalEINTRTest) ... ok
test_sigwaitinfo (__main__.SignalEINTRTest) ... ok
test_accept (__main__.SocketEINTRTest) ... ok
test_open (__main__.SocketEINTRTest) ... ok
test_os_open (__main__.SocketEINTRTest) ... ok
test_recv (__main__.SocketEINTRTest) ... ok
test_recvmsg (__main__.SocketEINTRTest) ... ok
test_send (__main__.SocketEINTRTest) ... ok
test_sendall (__main__.SocketEINTRTest) ... ok
test_sendmsg (__main__.SocketEINTRTest) ... ok
test_sleep (__main__.TimeEINTRTest) ... ok

----------------------------------------------------------------------
Ran 24 tests in 6.436s

OK (skipped=2)
Timeout (0:15:00)!
Thread 0x00007f0272748700 (most recent call first):
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/connection.py", line 379 in _recv
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/connection.py", line 414 in _recv_bytes
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/connection.py", line 250 in recv
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/pool.py", line 599 in _handle_results
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/threading.py", line 882 in run
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/threading.py", line 944 in _bootstrap_inner
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/threading.py", line 902 in _bootstrap

Thread 0x00007f027a8a6500 (most recent call first):
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/pool.py", line 674 in _help_stuff_finish
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/pool.py", line 689 in _terminate_pool
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/util.py", line 201 in __call__
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/pool.py", line 656 in terminate
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/_test_multiprocessing.py", line 2521 in test_terminate
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/case.py", line 616 in _callTestMethod
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/case.py", line 659 in run
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/case.py", line 719 in __call__
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/suite.py", line 122 in run
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/suite.py", line 84 in __call__
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/suite.py", line 122 in run
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/suite.py", line 84 in __call__
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/suite.py", line 122 in run
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/suite.py", line 84 in __call__
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/runner.py", line 176 in run
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/support/__init__.py", line 2032 in _run_suite
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/support/__init__.py", line 2128 in run_unittest
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/runtest.py", line 209 in _test_module
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/runtest.py", line 234 in _runtest_inner2
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/runtest.py", line 270 in _runtest_inner
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/runtest.py", line 140 in _runtest
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/runtest.py", line 193 in runtest
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/runtest_mp.py", line 73 in run_tests_worker
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/main.py", line 654 in _main
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/main.py", line 634 in main
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/main.py", line 712 in main
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/regrtest.py", line 43 in _main
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/regrtest.py", line 47 in <module>
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/runpy.py", line 85 in _run_code
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/runpy.py", line 192 in _run_module_as_main
Timeout (0:15:00)!
Thread 0x00007f251d553700 (most recent call first):
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/connection.py", line 379 in _recv
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/connection.py", line 414 in _recv_bytes
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/connection.py", line 250 in recv
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/pool.py", line 599 in _handle_results
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/threading.py", line 882 in run
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/threading.py", line 944 in _bootstrap_inner
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/threading.py", line 902 in _bootstrap

Thread 0x00007f2523e76500 (most recent call first):
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/pool.py", line 674 in _help_stuff_finish
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/pool.py", line 689 in _terminate_pool
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/util.py", line 201 in __call__
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/multiprocessing/pool.py", line 656 in terminate
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/_test_multiprocessing.py", line 2521 in test_terminate
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/case.py", line 616 in _callTestMethod
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/case.py", line 659 in run
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/case.py", line 719 in __call__
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/suite.py", line 122 in run
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/suite.py", line 84 in __call__
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/suite.py", line 122 in run
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/suite.py", line 84 in __call__
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/suite.py", line 122 in run
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/suite.py", line 84 in __call__
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/unittest/runner.py", line 176 in run
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/support/__init__.py", line 2032 in _run_suite
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/support/__init__.py", line 2128 in run_unittest
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/runtest.py", line 209 in _test_module
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/runtest.py", line 234 in _runtest_inner2
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/runtest.py", line 270 in _runtest_inner
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/runtest.py", line 153 in _runtest
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/runtest.py", line 193 in runtest
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/main.py", line 318 in rerun_failed_tests
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/main.py", line 691 in _main
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/main.py", line 634 in main
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/libregrtest/main.py", line 712 in main
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/test/__main__.py", line 2 in <module>
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/runpy.py", line 85 in _run_code
  File "/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.pgo/build/Lib/runpy.py", line 192 in _run_module_as_main
make: *** [buildbottest] Error 1

pablogsal · 2019-10-13T15:41:26Z

The buildbot failure is very likely unrelated, I am investigating.

bpo-38202: PyObject_GetIter() should be checked for failure

6b0388d

An exception may occur during a PyObject_GetIter() call.

ZackerySpytz requested a review from methane as a code owner Sep 17, 2019

the-knights-who-say-ni added the CLA signed label Sep 17, 2019

bedevere-bot added the awaiting review label Sep 17, 2019

corona10 requested changes Oct 8, 2019

View changes

bedevere-bot added awaiting core review and removed awaiting review labels Oct 8, 2019

methane approved these changes Oct 9, 2019

View changes

bedevere-bot added awaiting merge and removed awaiting core review labels Oct 9, 2019

corona10 added the skip news label Oct 10, 2019

Add unit test

2f81683

serhiy-storchaka added 2 commits Oct 10, 2019

Update test_dictviews.py

b5a9eab

Merge branch 'master' into bpo-38202-test_dictviews-fatal-error

1e4a4e6

serhiy-storchaka approved these changes Oct 10, 2019

View changes

Fix indentation.

d8d67b0

serhiy-storchaka merged commit b16e382 into python:master Oct 13, 2019

bedevere-bot removed the awaiting merge label Oct 13, 2019

jacobneiltaylor pushed a commit to jacobneiltaylor/cpython that referenced this pull request Dec 5, 2019

bpo-38202: Fix a crash in dict_view & non-itearble. (pythonGH-16241)

e4817f3

bpo-38202: PyObject_GetIter() should be checked for failure #16241

bpo-38202: PyObject_GetIter() should be checked for failure #16241

ZackerySpytz commented Sep 17, 2019 •

edited by bedevere-bot

serhiy-storchaka commented Sep 17, 2019

ZackerySpytz commented Oct 4, 2019

serhiy-storchaka commented Oct 4, 2019

corona10 left a comment

corona10 commented Oct 9, 2019

pablogsal commented Oct 10, 2019

pablogsal commented Oct 10, 2019

serhiy-storchaka commented Oct 10, 2019

bedevere-bot commented Oct 13, 2019

pablogsal commented Oct 13, 2019

bpo-38202: PyObject_GetIter() should be checked for failure #16241

bpo-38202: PyObject_GetIter() should be checked for failure #16241

Conversation

ZackerySpytz commented Sep 17, 2019 • edited by bedevere-bot

serhiy-storchaka commented Sep 17, 2019

ZackerySpytz commented Oct 4, 2019

serhiy-storchaka commented Oct 4, 2019

corona10 left a comment

corona10 commented Oct 9, 2019

pablogsal commented Oct 10, 2019

pablogsal commented Oct 10, 2019

serhiy-storchaka commented Oct 10, 2019

bedevere-bot commented Oct 13, 2019

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

pablogsal commented Oct 13, 2019

ZackerySpytz commented Sep 17, 2019 •

edited by bedevere-bot