Sitelet https://github.com/python/cpython/issues/140209
Skip to content

TSAN reports multiple data races, SEGV errors, and thread leaks when running the full test suite #140209

Description

@ashm-dev

Bug report

Bug description:

Description:

When running the full test suite (./python -m test) on a debug build of the main branch configured with ThreadSanitizer (TSAN) and the experimental JIT, the test run terminates with a FAILURE result.

The attached logs show thousands of warnings from TSAN, including:

  1. Massive numbers of data race warnings, primarily during the parallel initialization of sub-interpreters.
  2. Several DEADLYSIGNAL (SEGV) reports, causing processes to abort.
  3. Multiple thread leak warnings related to multiprocessing tests.
  4. Failures in several core tests, including test_datetime, test_faulthandler, test_pyrepl, and test.test_multiprocessing_forkserver.test_manager.

Build & Environment

  • CPython Version: 3.15.0a1+ (heads/main:7ac94fcb1d0, Oct 16 2025)
  • Platform: Linux-6.12.48+deb13-cloud-amd64-x86_64-with-glibc2.41
  • Compiler: Clang 19.1.7
  • Configuration command:
    CC=clang CXX=clang++ ./configure --with-thread-sanitizer --with-pydebug --enable-experimental-jit=yes
  • Build command:
    make -j$(nproc)
  • Test command:
    ./python -m test -j$(nproc)

Summary of Failures

The test suite concludes with the following result:

== Tests result: FAILURE ==

4 tests failed:
    test.test_multiprocessing_forkserver.test_manager test_datetime
    test_faulthandler test_pyrepl

Key Issues Reported by ThreadSanitizer

The full logs will be attached to this report, but here are the primary categories of errors observed.

1. Data Races during Sub-interpreter Initialization

Thousands of data race warnings are reported when multiple threads attempt to initialize global state concurrently. This is especially prominent in test_datetime, which fails with an exit code from TSAN.

  • Example Data Race Warning:
    ==================
    WARNING: ThreadSanitizer: data race (pid=1111680)
      Write of size 8 at 0x56174f42e490 by thread T2:
        #0 _PyExc_InitTypes /root/cpython/main/Objects/exceptions.c:4428:32
        #1 pycore_init_types /root/cpython/main/Python/pylifecycle.c:732:9
        ...
    
      Previous write of size 8 at 0x56174f42e490 by thread T1:
        #0 _PyExc_InitTypes /root/cpython/main/Objects/exceptions.c:4428:32
        #1 pycore_init_types /root/cpython/main/Python/pylifecycle.c:732:9
        ...
    
      Location is global '_PyExc_ReferenceError' of size 416 at 0x56174f42e300 (python+0xa57490)
    ...
    SUMMARY: ThreadSanitizer: data race /root/cpython/main/Objects/exceptions.c:4428:32 in _PyExc_InitTypes
    ==================
    
  • Resulting test_datetime Failure:
    test test_datetime failed -- Traceback (most recent call last):
    ...
    AssertionError: Process return code is 66
    command line: ['/root/cpython/main/python', '-X', 'faulthandler', '-I', '-c', "if True:\n from concurrent.futures import InterpreterPoolExecutor\n\n def func():\n import _datetime\n ..."]
    

2. SEGV Errors and Process Aborts

Multiple tests trigger fatal segmentation faults, which are caught and reported by TSAN.

  • Example SEGV Report:
    ThreadSanitizer:DEADLYSIGNAL
    ==1101487==ERROR: ThreadSanitizer: SEGV on unknown address 0x00000010ceaf (pc 0x7f05f669195c bp 0x000000000000 sp 0x7ffde6ad06a0 T1101487)
    ==1101487==The signal is caused by a READ memory access.
        #0 <null> <null> (libc.so.6+0x9495c)
        #1 raise <null> (libc.so.6+0x3fcc1)
        #2 raise <null> (python+0x103503)
        #3 faulthandler_raise_sigsegv /root/cpython/main/./Modules/faulthandler.c:1158:5
        ...
    SUMMARY: ThreadSanitizer: SEGV (/lib/x86_64-linux-gnu/libc.so.6+0x9495c) 
    ==1101487==ABORTING
    

3. Thread Leaks

TSAN reports that threads created during multiprocessing tests are not properly joined, leading to resource leaks.

  • Example Thread Leak Warning:
    ==================
    WARNING: ThreadSanitizer: thread leak (pid=1106511)
      Thread T2 'Thread-2 (handl' (tid=1106562, finished) created by thread T1 at:
        #0 pthread_create <null> (python+0xfaa35)
        #1 do_start_joinable_thread /root/cpython/main/Python/thread_pthread.h:281:14
        ...
    SUMMARY: ThreadSanitizer: thread leak (/root/cpython/main/python+0xfaa35) in pthread_create
    ==================
    

CPython versions tested on:

3.15, CPython main branch

Operating systems tested on:

Linux

Activity

  1. ashm-dev commented on Oct 16, 2025

    @ashm-dev
    ContributorAuthor
  2. StanFromIreland commented on Oct 16, 2025

    @StanFromIreland
    Member

    In our CI, we only run a subset of tests (--tsan):

    TSAN_TESTS = [
        'test_asyncio',
        'test_capi',
        'test_code',
        'test_ctypes',
        'test_concurrent_futures',
        'test_enum',
        'test_functools',
        'test_httpservers',
        'test_imaplib',
        'test_importlib',
        'test_io',
        'test_logging',
        'test_opcache',
        'test_queue',
        'test_signal',
        'test_socket',
        'test_sqlite3',
        'test_ssl',
        'test_syslog',
        'test_thread',
        'test_thread_local_bytecode',
        'test_threadedtempfile',
        'test_threading',
        'test_threading_local',
        'test_threadsignals',
        'test_weakref',
        'test_free_threading',
    ]
    

    So I consider this a feature.

  3. added
    type-featureA feature request or enhancement
    testsTests in the Lib/test dir
    and removed
    type-bugAn unexpected behavior, bug, or error
    on Oct 16, 2025
  4. changed the title [-]TSAN reports multiple data races, SEGV errors, and thread leaks when running the test suite on main branch[/-] [+]TSAN reports multiple data races, SEGV errors, and thread leaks when running the full test suite on main branch[/+] on Oct 16, 2025
  5. changed the title [-]TSAN reports multiple data races, SEGV errors, and thread leaks when running the full test suite on main branch[/-] [+]TSAN reports multiple data races, SEGV errors, and thread leaks when running the full test suite[/+] on Oct 16, 2025
  6. colesbury commented on Oct 16, 2025

    @colesbury
    Contributor
    1. See Various data races in subinterpreter tests under TSAN #129824
    2. We run tests with TSAN_OPTIONS=handle_segv=0. Some test cases that are expected to crash launch processes with TSAN_OPTIONS=handle_segv=0 in case you forget it when running TSAN locally. If some test is missing that, we can add it.
    3. Some tests that use daemon threads leak threads. That's expected. We suppress that warning in https://github.com/python/cpython/blob/main/Tools/tsan/suppressions.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testsTests in the Lib/test dirtype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions