Sitelet https://github.com/Netis/cloud-probe/issues/294
Skip to content

cpworker: log lines from concurrent threads interleave (log_log writes a line in three stdio calls; localtime is not thread-safe) #294

Description

@vaderyang

Summary

log_log (cpworker/src/log.c, 0.9.x) is called from several threads, among them the main thread, the unix control thread, the output thread and the capture threads. It is not thread-safe in two ways:

  1. A line is written with three separate stdio calls: fprintf for the prefix, vfprintf for the message and fprintf for "\n". stdio locks each call, not the line, so lines from concurrent threads interleave. In a stress test, 4 threads × 20,000 log_log calls against 0.9.x log.c corrupted 29,193 of 80,000 lines:
    2026-10-02T21:16:39 INFO  thread.c:3: 2026-10-02T21:16:39 INFO  thread.c:1: thread=1 seq=1 payload=…
    2026-10-02T21:16:39 INFO  thread.c:0: thread=0 seq=4 payload=…thread=2 seq=2 payload=…thread=1 seq=2 payload=…
    
  2. localtime() returns a pointer to a static struct tm, which strftime reads while other threads overwrite it. ThreadSanitizer reports this as a data race: log_log (log.c:43) from task_manager_new races with unix_client_recv in the control thread and with task_manager_output_loop.

Impact

cpdaemon splits the worker's stderr into lines and parses each one (parseLogLine, cpdaemon/pkg/worker/log.go) before forwarding it to its own log and to CPM. An interleaved line is handled in one of two ways:

  • it is parsed with the level of the first prefix, so another thread's ERROR can be forwarded as INFO, with a second prefix inside the message;
  • it is reported as invalid cpworker log, for a fragment that has no prefix.

Threads log concurrently mainly at task start, on reload, on control requests and on error paths, which is exactly when the log matters.

Expected

Each log line is written atomically. For example, format it into one buffer, then issue a single fputs/write, or wrap the three calls in flockfile(stderr) / funlockfile(stderr). Use localtime_r.

Reproduction

Build the program below against cpworker/src/log.c (gcc -pthread -Icpworker/src main.c cpworker/src/log.c), then run ./a.out 2>out.txt. Count the lines that do not match ^<time> INFO thread.c:N: thread=N seq=N payload=…$. For the TSan report, build the unit tests with -fsanitize=thread and run control_inproc under setarch -R.

#include <pthread.h>
#include "log.h"
static void *run(void *arg) {
    long id = (long)arg;
    for (int i = 0; i < 20000; i++)
        log_log(LOG_INFO, "thread.c", (int)id, "thread=%ld seq=%d payload=abcdefghijklmnopqrstuvwxyz", id, i);
    return NULL;
}
int main(void) {
    pthread_t t[4];
    for (long i = 0; i < 4; i++) pthread_create(&t[i], NULL, run, (void *)i);
    for (int i = 0; i < 4; i++) pthread_join(t[i], NULL);
}
中文原文

log_log 被主线程、控制线程、输出线程、抓包线程并发调用,但它不是线程安全的:(1) 一行日志由 3 次 stdio 调用写出(前缀、消息、换行),stdio 只对单次调用加锁,多线程下各行相互交错,4 线程 × 2 万次压测中 80000 行有 29193 行损坏;(2) localtime() 返回静态 struct tm,strftime 读取时可能被其他线程覆盖,TSan 报告 log.c:43 的数据竞争。cpdaemon 按行解析 worker 的 stderr 并转发到 CPM,交错行会以错误的级别转发(其他线程的 ERROR 变成 INFO)或被记为 invalid cpworker log。并发日志主要出现在任务启动、reload、控制请求和错误路径上,正是日志最重要的时候。期望每行原子写出(单次写入或 flockfile),并使用 localtime_r。

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions