Repository navigation
Add os.statx() function #83714
Description
Activity
alexander255 commented
on Feb 3, 2020 alexander255mannequinMannequinAuthorBackground: For a long time several Linux filesystems have been tracking two extra bits of information. The file attributes bits1 and the file creation time (aka crtime aka btime aka birthtime)2. Before Linux 4.11 accessing these required secret knowledge (ioctl numbers) or access to unstable interfaces (debugfs). However since that version the statx(2) system call3 has finally been added (it has a long history), which exposes these two fields adds (struct) space for potentially more.
Since CPython already exposes
st_birthtimeon FreeBSD and friends, I think it would be fair to also expose this field on Linux. As the timestamp value is only available on some file systems and configurations it is not guaranteed that the system call will return a value for btime at all. I suppose the field should be set toNonein that case. In my opinion it should also become a regular field (available on all platforms) since, with this addition, we now have a suitable value to return on every major platform CPython targets:stx_btimeon Linux,st_birthtimeon macOS/FreeBSD andst_ctimeon Windows.stx_attributescould be exposed as a newst_attributesflag specific to Linux as there is no equivalent on other platforms to my knowledge (Window'sst_file_attributesis similar in some aspects but has a completely different format and content).There is a Python script I created, that calls statx(2) using ctypes here: https://github.com/ipfs/py-datastore/blob/e566d40a8ca81d8628147e255fe7830b5f928a43/datastore/filesystem/util/statx.py
It may be useful as a reference when implementing this in C.Reacted by Jeffrey Bosboom and David L. QiuThe statx call was introduced by Kernel 4.11 in 2017. Major LTS Linux distributions like Debian 9, Ubuntu 16.04, and CentOS 7 use older Kernels like Linux 4.9 LTS or 3.10 LTS.
In general we try to support older Kernel ABIs even when Python is compiled on a system with more recent ABI. This means you have to perform a runtime feature detection and fall back to old stat when the syscall fails.
alexander255 commented
on May 17, 2020 alexander255mannequinMannequinAuthorI thought this might be the case, I'll look into adapting the patch accordingly then.
You can find an example in Python/bootstrap_hash.c that deals with getrandom syscall.
I have rebased the associated PR for Python 3.12-dev and added backward-compatible support for new
statxfeatures introduced in Linux 5.8+. Is thestatfallback still needed?Also: What about linking with the
statxC library symbol on systems that have it? Should it use a weak reference ordlopeninstead to ensure that the Python build will work with an older C library? Or should it only unconditionally use thesyscallfunction (in which case all the header definitions will need to be duplicated as well or it would still require a newer system to build)?I'm interested in this, is there some progress since?
I'm interested in this, is there some progress since?
The PR is outdated. Someone has to update it.
I'm interested in working on this. I've posted a draft PR as earnest effort (in the sense of earnest money), but we should agree on the answers to these questions before serious review.
Do we fall back to stat?
The previous PR gh-19125 did not fall back to stat if statx was unavailable. A comment in this issue objected to this because statx was relatively new at the time. statx was introduced in Linux 4.11, released April 2017. The oldest currently-supported upstream LTS branch is 5.4, released November 2019. If I am reading the RHEL life cycle and RHEL release dates correctly, the oldest supported release is RHEL8, based on kernel 4.18. It is likely that all mainstream Linux distributions ship kernels supporting statx. On the other hand, there are probably embedded environments stuck on ancient kernels; do their users want to run the newest Python?
A complicating factor is default-deny seccomp syscall filters that have not been updated to allow statx. If they deny with ENOSYS and we fall back to stat, Python will keep working under those filters; if we don't fall back, Python will almost certainly fail during startup.
The code to fall back is simple enough, but testing the fallback path would require running on an old kernel or under a seccomp filter.
My draft PR has a fallback, but no tests for the fallback.
How do we call statx?
In order of increasing reliance on the build environment, here are the ways we can call statx:
-
Hardcode the syscall numbers for architectures we care about, provide our own definition of struct statx and related flags, and call via syscall(). Build is self-sufficient.
-
Use syscall numbers from sys/syscall.h, definitions of struct stat and flags from linux/stat.h, provide our own definitions of flags newer than statx itself, call via syscall(). Build requires Linux kernel userspace API headers >= 4.11. (New userspace API headers can be used on old kernels; they're distinct from the headers used to build kernel modules.)
-
Call using libc's statx wrapper (weakly linked), provide our own definitions of flags newer than statx itself. Requires glibc version 2.28 (August 2018) at build time and run time (but can fall back to stat if runtime libc is old).
My draft PR is option 3.
How do we indicate missing information?
We won't always get everything we ask for in struct statx, depending on the kernel, filesystem type, and filesystem instance (e.g., ext4 supports btime, but really old ext4 filesystems may not). For btime specifically, st_birthtime has never been exposed on Linux, so it's tempting to set it to None when it isn't available. Code that checks the platform before using st_birthtime and code using
getattr(st, 'st_birthtime', None)will work, but code that relies on catching AttributeError may not expect st_birthtime to be None -- and there's an example of the latter in test_os.py. So we have to fill it in, either with 0 or with whatever statx returned (not necessarily 0).The basic stats (also in struct stat) are always initialized, but their bits may not be set in stx_mask. linux/stat.h gives CIFS as an example that might clear STATX_UID/GID if they're being overridden client-side, though I haven't observed this. I have observed a CIFS mount on Linux 6.15.4 that doesn't set STATX_ATIME. We obviously can't set st_atime/uid/gid to None.
I think we should indicate to the application what fields are and aren't reliable, but I don't know how. Exposing stx_mask isn't extensible to other operating systems. We could make up our own mask and flags. We could have a set-like member for tests of the form
'st_birthtime' in st.present, though I don't know how to implement that performantly (and stat is likely hot in some programs). We could introduce int/float subclasses with an extra 'present' member, but I'm sure there are programs that expect exactly int/float. I don't like any of these options.My draft PR does nothing here.
Which mount ID do we want?
There are two flags to ask for mount IDs, STATX_MNT_ID and STATX_MNT_ID_UNIQUE. The non-unique ID indexes
/proc/self/mountinfoand is subject to reuse race conditions; the unique ID can be used with the statmount system call, which requires CAP_SYS_ADMIN and does not currently have a glibc wrapper. Unfortunately they both initialize stx_mnt_id, so we can only get one or the other (unless we call statx twice). Setting both flags is not an error, but I do not think the result is defined; my experiments have always obtained the unique ID.My draft PR provides the non-unique ID because I think it will be more useful to general Python programs.
How do we test the new members?
ntfs-3g and cifs allow setting btime via the
user.ntfs/cifs.creationtimemagic xattr. I don't know offhand if ntfs-3g reports btime (being a FUSE filesystem). cifs does, but that requires a server to be available. We could also just create a file, wait, modify the file, and check its btime differs from its mtime and ctime.We can test stx_mnt_id by looking for it in
/proc/self/mountinfo.Only btrfs and bcachefs currently support stx_subvol. bcachefs has an uncertain future, so we'd want btrfs.
Of the stx_attributes, only STATX_ATTR_MOUNT_ROOT is easily testable, by opening
/proc/self/mountinfoand checking the flag is set on mount points. This flag should also be independent of both filesystems.My draft PR adds no new tests.
Do we want to map attributes to st_flags?
The documentation for the os module claims st_flags "may be available on some Unix systems (such as Linux)", but as far as I can tell it has never been exposed on Linux. The previous PR gh-19125 mapped both Windows and Linux attribute bits to st_flags. Do we want that? Do we want to map st_flags back to attribute bits? (I don't know which systems currently have st_flags or how they are used. Many of them are macOS-specific. The os.chflags documentation says "Availability: Unix, not WASI" but it's not present on Linux. BSD?)
My draft PR doesn't do this.
What do we name stx_attributes?
My PR names it st_attributes, but maybe st_linux_attributes is safer.
Future work
statx supports the AT_STATX_FORCE_SYNC and AT_STATX_DONT_SYNC flags to ask for up-to-date or cached information. (AT_STATX_SYNC_AS_STAT is 0.) I'd like to expose this as a sync=True/False/None argument to os.stat, but I need to see what other operating systems offer first. In particular, the statx flags are hints; if cached info is not available, statx will hit the network/tape/etc. I can imagine that other operating systems might have a flag to say "cached if possible, else fail, never block", which would have to be handled differently.
-
@jbosboom: Have you looked at my PR before writing yours?
st_birthtimemay be NoneI’ve addressed the “
st_birthtimemay be None” issue by only exposing the new information asst_btimeand addingst_birthtimeas legacy alias on BSD only, so existing code checking for the presence ofst_birthtimeshould not be affected (but will continue to only detect btime on BSD). I’ve also copied the value ofst_ctimeon Windows tost_btimesince file creation time is exposed asst_ctimethere (which also continues to work and be inconsistent across platforms).So my patch
st_btimeis always defined but may beNonein some cases on Linux – this is fully backwards-compatible and solves all issues.This should also solve all cases of missing information in practise and set at precedent for future fields.
Extra fields
I didn’t do this, but they should always be exposed on Linux only, but are allowed to be
None.Do we want to map attributes to st_flags?
I had implemented mapping these for Linux based on BSD semantics IIRC.
Perhaps I did something about
st_attributesas well, but I cannot remember anymore…70 remaining items
- added 11 commits that reference this issue
on Dec 6, 2025
statxon more recent Linux to exposest_flagsandst_btimeon all platforms #19125Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs