public inbox for gentoo-catalyst@lists.gentoo.org
 help / color / mirror / Atom feed
From: Matt Turner <mattst88@gentoo.org>
To: "Michael 'veremitz' Everitt" <gentoo@veremit.xyz>
Cc: gentoo-catalyst@lists.gentoo.org
Subject: Re: [gentoo-catalyst] [PATCH 19/21] catalyst: Set jobs/load-average via catalyst.conf
Date: Wed, 20 May 2020 18:19:01 -0700	[thread overview]
Message-ID: <CAEdQ38Ef=FfjAACnktoGhEuC96qx842+6Swwr+HP9bXKS81m=Q@mail.gmail.com> (raw)
In-Reply-To: <9d529c77-14b3-cb7b-f2ee-32b4bbc2f464@veremit.xyz>

On Wed, May 20, 2020 at 5:37 PM Michael 'veremitz' Everitt
<gentoo@veremit.xyz> wrote:
>
> On 21/05/20 01:18, Matt Turner wrote:
> > On Wed, May 20, 2020 at 4:00 PM Michael 'veremitz' Everitt
> > <gentoo@veremit.xyz> wrote:
> >> On 20/05/20 04:42, Matt Turner wrote:
> >>> We currently have two mechanisms of setting MAKEOPTS: in spec files and
> >>> in catalystrc.
> >>>
> >>> Setting makeopts in spec files doesn't make sense. The spec should
> >>> describe the thing that's being built and not contain options that are
> >>> specific to the build system.
> >>>
> >>> Setting makeopts via catalystrc is better, but it only applies to the
> >>> actual build system invocations, leaving emerge to run jobs serially or
> >>> again requiring configuration specific to the build machine to be put
> >>> into the spec file. For example:
> >>>
> >>>       update_seed_command: ... --jobs 5 --load-average 5
> >>>
> >>> With jobs and load-average specified in catalyst.conf, catalyst has the
> >>> information required to configure both emerge and the build systems
> >>> emerge executes.
> >>>
> >>> This removes the undocumented makeopts spec file option and replaces it
> >>> with jobs and load-average settings in catalyst.conf.
> >>>
> >>> Signed-off-by: Matt Turner <mattst88@gentoo.org>
> >>> ---
> >>>  catalyst/base/stagebase.py          | 12 +++++-------
> >>>  catalyst/defaults.py                |  2 ++
> >>>  doc/catalyst-config.5.txt           | 15 ++++++++++++---
> >>>  etc/catalyst.conf                   |  8 ++++++++
> >>>  etc/catalystrc                      |  3 ---
> >>>  targets/support/chroot-functions.sh |  8 ++++++++
> >>>  6 files changed, 35 insertions(+), 13 deletions(-)
> >> NACK. This and patch 20 make it impossible to customise specs for different
> >> arches/subarches/libc/etc using different distcc_hosts with varying numbers
> >> of hosts and/or cores available. In many cases, you may not be able to set
> >> up identical distcc 'farms' or 'clusters' with the complete set of
> >> toolchains/chroots applicable to a 'standard' installation, due to hardware
> >> limitations.
> > I don't follow. You mention varying distcc hosts per arches/subarches/libc.
> >
> > Let's say there are 4 systems on the network. 3x amd64 systems and 1
> > mips system.
> >
> > If I'm compiling for amd64 on system and I'm using distcc, there's a
> > fixed number of distcc hosts on the network, aren't there?
> Yes, but do you feed your amd64 jobs to your mips system? how do you filter
> between them?

I wouldn't... but I don't think that matters? You just set
distcc_hosts to not send jobs to that system.

> > If I'm building for mips on the mips system with cross compilers
> > installed on, say two of the amd64 systems, then there's a fixed
> > number of distcc hosts on the network, aren't there?
> >
> > So, I don't think arches (or subarches) has any bearing.
> >
> > And for different libcs, distcc doesn't have any bearing on that, does
> > it? distcc (without pump mode, which has been removed from Gentoo)
> > sends preprocessed source files across the network, so what libc you
> > have doesn't have any bearing either.
> I have one or two systems with musl toolchains, but not all, again, how do
> I choose between them, from a common build host, for each different spec
> file variant I wish to run?

Like I said, distcc sends a preprocessed file across the network, so
there's not a dependence on the libc. I.e., a musl system running
distccd can compile files just fine for a system running glibc.

> > You seem to be thinking that setting a single system-wide distcc_hosts
> > is somehow limiting, but I can't see how. Please explain.
> >
> I fail to see how you can select from a pool of distcc slaves for a given
> job type/configuration.
>
> I'm assuming you've used distcc before? With cluster project's help I
> actually have several distcc daemons running on a given host, if I should
> require eg. gcc-8, 9 or (in the near future) 10. These run on different
> ports on each slave system.
> How would you propose selecting between these separate daemons for a given
> build in your new schema?

Yes, I use distcc to cross compile for slow mips and arm systems.


I don't understand why it takes so much more effort on the part of the
receiver to interpret your messages, but good grief.

I think the case Michael is thinking of is, e.g. a system building
both rel_type: default and rel_type: hardened stages. The set of
distcc slaves that have a non-hardened toolchain might be different
from the set of distcc slaves with a hardened toolchain.

I'm unsure how this works, so I'm asking #gentoo-toolchain.

So all of the previous email exchange was a collection of red herrings...


  parent reply	other threads:[~2020-05-21  1:19 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-20  3:42 [gentoo-catalyst] [PATCH 01/21] targets: Remove copy_{file,symlink,lib} functions Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 02/21] targets: Remove check_genkernel_version() function Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 03/21] catalyst: Drop --cli option Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 04/21] catalyst: Remove PythonDir setting Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 05/21] catalyst: Switch internal snapshot option parsing to SpecParser Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 06/21] catalyst: Remove unused decompression_mode spec option Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 07/21] catalyst: Delete unused {required,valid}_build_targets Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 08/21] catalyst: Disallow DEBUG and VERBOSE in spec files Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 09/21] catalyst: Add decompressor_search_order as valid spec option Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 10/21] catalyst: Remove spec file options from valid_config_file_values Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 11/21] catalyst: Add a missing options to valid_config_file_values Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 12/21] catalyst: Disallow config file options in spec files Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 13/21] catalyst: Remove support for source_matching="loose" Matt Turner
2020-05-20 22:47   ` Brian Dolbec
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 14/21] catalyst: Convert catalyst.conf to TOML Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 15/21] catalyst: Verify config options against valid_config_file_values Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 16/21] catalyst: Don't even try to make envars from dicts Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 17/21] catalyst: Only emit true boolean envars Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 18/21] catalyst: Support emitting int/float envars Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 19/21] catalyst: Set jobs/load-average via catalyst.conf Matt Turner
     [not found]   ` <94424974-cfe8-9766-8712-ae6fa8bbf825@veremit.xyz>
2020-05-21  0:18     ` Matt Turner
     [not found]       ` <9d529c77-14b3-cb7b-f2ee-32b4bbc2f464@veremit.xyz>
2020-05-21  1:19         ` Matt Turner [this message]
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 20/21] catalyst: Configure distcc_hosts in the config file Matt Turner
2020-05-20  3:42 ` [gentoo-catalyst] [PATCH 21/21] catalyst: Drop ConfigParser Matt Turner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAEdQ38Ef=FfjAACnktoGhEuC96qx842+6Swwr+HP9bXKS81m=Q@mail.gmail.com' \
    --to=mattst88@gentoo.org \
    --cc=gentoo-catalyst@lists.gentoo.org \
    --cc=gentoo@veremit.xyz \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox