| 1 |
On Fri, 6 Mar 2020 16:06:00 +0800 |
| 2 |
heroxbd@g.o wrote: |
| 3 |
|
| 4 |
> From: Benda Xu <heroxbd@g.o> |
| 5 |
> |
| 6 |
> If ghc spawns too many C compilers, it will exhaust file descripters. |
| 7 |
|
| 8 |
I don't think ghc spawns more than 1 parallel gcc per compiled haskell file. |
| 9 |
I'd expect a small constant overhead of file descriptors per compilation thread, |
| 10 |
something like 5 fds (not 16 as your '64' limit implies): |
| 11 |
|
| 12 |
- 2 opened files for write a result (one .hi and one .o file) |
| 13 |
- 3 file descriptors per external tool to handle pipe std{in,out,err}. |
| 14 |
|
| 15 |
Unless it's a -split-objs effect (which I don't believe is done in parallel), then |
| 16 |
the FD limit is orthogonal to job count. |
| 17 |
|
| 18 |
Running 'strace -f -y' against `ghc --make` invocation should make it obvious. |
| 19 |
|
| 20 |
Please file a bug to find out where these file descriptors come from. |
| 21 |
|
| 22 |
Once we understand better where descriptors come from the bug might |
| 23 |
need to go upstream eventually to more explicitly manage scarce fd resources |
| 24 |
on the side of compilation manager. There should be no reason to limit parallelism |
| 25 |
as long as there is no external process forking. |
| 26 |
|
| 27 |
> In the reference, it was thought to be a macOS bug for aggressive fd |
| 28 |
> limits. But the ghc bug also applies to GNU/Linux, when ghc is |
| 29 |
> asked to spawn, for example 256, jobs. |
| 30 |
> |
| 31 |
> This patch circumvents this ghc design flaw. |
| 32 |
|
| 33 |
I don't see it as a design flaw. |
| 34 |
|
| 35 |
> Reference: https://github.com/commercialhaskell/stack/issues/1177 |
| 36 |
> Reference: https://github.com/commercialhaskell/stack/issues/1979 |
| 37 |
> Signed-off-by: Benda Xu <heroxbd@g.o> |
| 38 |
> --- |
| 39 |
> eclass/ghc-package.eclass | 4 +++- |
| 40 |
> 1 file changed, 3 insertions(+), 1 deletion(-) |
| 41 |
> |
| 42 |
> diff --git a/eclass/ghc-package.eclass b/eclass/ghc-package.eclass |
| 43 |
> index 5361f09af1e9..d729f4a407b4 100644 |
| 44 |
> --- a/eclass/ghc-package.eclass |
| 45 |
> +++ b/eclass/ghc-package.eclass |
| 46 |
> @@ -203,7 +203,9 @@ ghc-make-args() { |
| 47 |
> # https://ghc.haskell.org/trac/ghc/ticket/9221#comment:57 |
| 48 |
> # SMP is a requirement for parallel GC's gen0 |
| 49 |
> # 'qb' balancing. |
| 50 |
> - echo "-j$(makeopts_jobs) +RTS -A256M -qb0 -RTS" |
| 51 |
> + local n=$(makeopts_jobs) |
| 52 |
> + [[ ${n} -gt 64 ]] && n=64 |
| 53 |
> + echo "-j${n} +RTS -A256M -qb0 -RTS" |
| 54 |
|
| 55 |
Needs an in-source comment why a limit is imposed. Otherwise looks ok to push |
| 56 |
while we are figuring out the limits. And a an explanation how it was calculated. |
| 57 |
|
| 58 |
> ghc_make_args=() |
| 59 |
> fi |
| 60 |
> echo "${ghc_make_args[@]}" |
| 61 |
> -- |
| 62 |
> 2.25.0 |
| 63 |
> |
| 64 |
> |
| 65 |
|
| 66 |
-- |
| 67 |
|
| 68 |
Sergei |
Archives