rmSync hits EPERM right after a clean server.stop() because a fire-and-forget icacls.exe still holds the directory
env paths · case
Symptom
Section titled “Symptom”A test suite that starts a server, stops it with await server.stop(true), and then removes its
scratch home fails in the teardown, not in the test:
error: EPERM: operation not permitted, rm 'D:\a\opencodex\opencodex\tests\.tmp-codex-accounts-test'error: EBUSY: resource busy or locked, rm 'C:\Users\RUNNER~1\AppData\Local\Temp\ocx-api-usage-esM6Go'Four Windows CI shards went red on every branch, including the released main, for three days.
Linux and macOS stayed green on the same commits. Individual test bodies passed; the
afterEach hook failed, and when the hook shared one fixed directory across cases, the first
failure poisoned every later case in the file (49 of 49 errors in one file, 377 in another were
hooks, not assertions).
// child.mjs — any child that touches the directory and outlives its parent's "done"import { spawn } from "node:child_process";import { mkdirSync, rmSync } from "node:fs";
mkdirSync("scratch", { recursive: true });// icacls holds the directory handle for the duration of the ACL rewritespawn("icacls.exe", ["scratch", "/inheritance:r", "/grant:r", "*S-1-5-32-544:(OI)(CI)F"], { stdio: "ignore" });// "shutdown" that forgot about the childrmSync("scratch", { recursive: true, force: true });// → EPERM on Windows; silently fine on Linux/macOSPS> node child.mjsnode:fs:... Error: EPERM: operation not permitted, rm 'scratch'Two Windows facts combine:
- File locking is mandatory. A handle opened without
FILE_SHARE_DELETE— which is what every ordinary process, includingicacls.exe, holds while it works on a directory — makes the kernel refuseunlink(EPERM) andrename(EBUSY) until the handle closes. POSIX only removes the name; the data lives until the last descriptor goes away, so the same code never notices there. - Nothing ties a child’s lifetime to its parent’s notion of “finished”. A
spawnwhose promise is dropped keeps running.server.stop()awaited listeners, background jobs and lifecycle hooks — everything the server started on purpose — but not the ACL flight a config read kicked off as an optimisation (hardenConfigDir()→hardenSecretDirAsync(), unawaited, introduced to stop the event loop from blocking on a slowicacls).
The regression was invisible on the machines developers use, and the CI signal was gated behind
workflow_dispatch, so it was attributed to “the hosted runner” for three days. It was the
product: the same suites had passed on the same windows-latest image before the async change.
Workaround
Section titled “Workaround”The shutdown contract has to own every child the process started. Scope the flight to the directory it works on and await it where you await everything else:
const flights = new Map<string, Promise<void>>();export async function flushConfigDirHardening(dir: string): Promise<void> { const flight = flights.get(dir); if (flight) await flight;}
// server.tsconst configDir = getConfigDir(); // capture BEFORE the flight startsserver.stop = async () => { await closeListeners(); await backgroundLifecycle.release(); await flushConfigDirHardening(configDir); // now rm after stop() is safe};Prove it with a test that holds the child on a promise and asserts stop() stays pending until
the promise resolves; drive it red once by commenting the await out.
Retrying rmSync on EPERM/EBUSY (50 × 50 ms) is a legitimate belt-and-braces for antivirus
and search-indexer handles, and this repo keeps one for fixtures. It is an unsafe primary fix
here: it hides the unowned child, the retry budget is a guess, and in production the same
child would still be holding a directory the uninstaller or a home move is about to touch.
This is the “who owns the child” half of the file-lifetime story. unlink-while-open-ebusy
is the “who owns the file” half; kill-hits-one-pid-or-the-whole-tree is what happens when the
parent leaves without either.