Sitelet https://github.com/nodejs/node/pull/65524
Skip to content

fs: improve performance of recursive directory read - #65524

Open
avivkeller wants to merge 1 commit into
nodejs:mainfrom
avivkeller:fs-perf
Open

fs: improve performance of recursive directory read#65524
avivkeller wants to merge 1 commit into
nodejs:mainfrom
avivkeller:fs-perf

Conversation

@avivkeller

@avivkeller avivkeller commented Aug 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Currently, when recursively reading a directory, we do a few things which can be considered slow:

  1. We round-trip between CPP and JS for each entry, meaning that a 5.4k-entry large directory needs to round-trip 5.4k times, after this PR, that's only one round trip.
  2. We stat'd every file previously, and now, we only stat when needed, using the native lstat over the JS version.

Benchmarks:

                                                                                             confidence improvement accuracy (*)    (**)   (***)
fs/bench-readdir-recursive.js withFileTypes='false' mode='callback' dir='lib' n=10                  ***    248.34 %      ±22.14% ±29.62% ±38.87%
fs/bench-readdir-recursive.js withFileTypes='false' mode='callback' dir='test/parallel' n=10        ***    471.49 %      ±45.94% ±61.85% ±81.99%
fs/bench-readdir-recursive.js withFileTypes='false' mode='promise' dir='lib' n=10                   ***    243.07 %      ±25.02% ±33.53% ±44.11%
fs/bench-readdir-recursive.js withFileTypes='false' mode='promise' dir='test/parallel' n=10         ***    485.55 %      ±37.61% ±50.61% ±67.05%
fs/bench-readdir-recursive.js withFileTypes='false' mode='sync' dir='lib' n=10                      ***    188.53 %      ±21.79% ±29.19% ±38.39%
fs/bench-readdir-recursive.js withFileTypes='false' mode='sync' dir='test/parallel' n=10            ***    497.94 %      ±31.68% ±42.61% ±56.39%
fs/bench-readdir-recursive.js withFileTypes='true' mode='callback' dir='lib' n=10                   ***    118.38 %      ±21.14% ±28.27% ±37.08%
fs/bench-readdir-recursive.js withFileTypes='true' mode='callback' dir='test/parallel' n=10         ***    290.39 %      ±21.57% ±28.89% ±37.99%
fs/bench-readdir-recursive.js withFileTypes='true' mode='promise' dir='lib' n=10                    ***     53.59 %      ±14.66% ±19.53% ±25.47%
fs/bench-readdir-recursive.js withFileTypes='true' mode='promise' dir='test/parallel' n=10                   8.72 %       ±9.27% ±12.33% ±16.06%
fs/bench-readdir-recursive.js withFileTypes='true' mode='sync' dir='lib' n=10                       ***     71.40 %      ±14.63% ±19.52% ±25.51%
fs/bench-readdir-recursive.js withFileTypes='true' mode='sync' dir='test/parallel' n=10             ***    269.65 %      ±26.21% ±35.19% ±46.45%

Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case, there are 12 comparisons, you can thus
expect the following amount of false-positive results:
  0.60 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.12 false positives, when considering a   1% risk acceptance (**, ***),
  0.01 false positives, when considering a 0.1% risk acceptance (***)

Also,
Fixes #58892

@nodejs-github-bot

nodejs-github-bot commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/performance

@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Aug 24, 2026
@avivkeller avivkeller added the fs Issues and PRs related to the fs subsystem / file system. label Aug 24, 2026
@avivkeller

avivkeller commented Aug 24, 2026

Copy link
Copy Markdown
Member Author

cc @nodejs/fs

@codecov

codecov Bot commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 71.92308% with 73 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.10%. Comparing base (54b4e37) to head (21bb129).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/node_file.cc 65.40% 45 Missing and 28 partials ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main   #65524      +/-   ##
==========================================
- Coverage   90.13%   90.10%   -0.04%     
==========================================
  Files         751      751              
  Lines      253635   253747     +112     
  Branches    47786    47822      +36     
==========================================
+ Hits       228615   228633      +18     
- Misses      16268    16356      +88     
- Partials     8752     8758       +6
Files with missing lines Coverage Δ
lib/fs.js 98.39% <100.00%> (-0.04%) ⬇️
lib/internal/fs/promises.js 92.45% <100.00%> (-0.10%) ⬇️
lib/internal/fs/utils.js 97.96% <100.00%> (+0.03%) ⬆️
src/node_file.cc 73.51% <65.40%> (-0.65%) ⬇️

... and 33 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@avivkeller avivkeller assigned avivkeller and unassigned avivkeller Aug 25, 2026
@avivkeller
avivkeller marked this pull request as draft August 25, 2026 03:53
@avivkeller
avivkeller marked this pull request as ready for review August 25, 2026 03:56
@avivkeller
fs: improve performance of recursive directory read
21bb129 
Signed-off-by: avivkeller <me@aviv.sh>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@anonrig anonrig Awaiting requested review from anonrig
@codebytere codebytere Awaiting requested review from codebytere

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to the fs subsystem / file system. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

all versions of readdir don't work in recursive mode when used with a buffer argument

2 participants

@avivkeller @nodejs-github-bot