April 27 [Issue 24524] New: Very slow process fork if RLIMIT_NOFILE is too high | ||||
---|---|---|---|---|
| ||||
https://issues.dlang.org/show_bug.cgi?id=24524 Issue ID: 24524 Summary: Very slow process fork if RLIMIT_NOFILE is too high Product: D Version: D2 Hardware: x86_64 OS: Linux Status: NEW Severity: enhancement Priority: P1 Component: phobos Assignee: nobody@puremagic.com Reporter: trikkuz@gmail.com In `std.process`, the `spawnProcessPosix` function, used by all functions that start a process, has a flag `Config.Flags.inheritFDs`. If set to false, it reads the maximum limit of files that can be opened by a process, `RFILE_NOFILE`, and then creates a structure for each one and finally iterates through them. Normally, the `RFILE_NOFILE` limit is set to 1024 or anyway a relatively low number. The hard limit depends on the system; in some, it is 2^20, in others even 2^30. When starting a container with Docker, the limit is raised to the maximum allowed, and this inevitably slows down the process visibly (because it has to allocate 2^20 (or 2^30) elements and then iterate through them. An user reported that on a system with 2^30 as the hard limit, it becomes literally unusable. There would be a faster way to do this by reading the open file descriptors from `/dev/fd` with the code below, but we are inside a `@nogc` block, so another solution must be found. ` import std.file : exists, isDir; if (exists("/dev/fd") && isDir("/dev/fd")) { import std.file : dirEntries, SpanMode; import std.path : baseName; import std.algorithm : map, filter; import std.array : array; import std.string : indexOfNeither; import std.conv : to; // List all file descriptors except stdin, stdout, stderr, and the pipe auto fds = dirEntries("/dev/fd", SpanMode.shallow) .map!(d => d.name.baseName) .filter!(d => d.indexOfNeither("0123456789") < 0) .map!(d => d.to!int) .filter!(d => d >=3 && d != forkPipeOut) .array; // Copy to avoid closing while iterating // Close only the file descriptors that are open. foreach(fd; fds) close(fd); } ` -- |
Copyright © 1999-2021 by the D Language Foundation