Re: [gentoo-catalyst] Couple of questions

[Chris Gianelloni]

On Thu, 2005-12-15 at 10:57 -0600, Mikey wrote:
> On Thursday 15 December 2005 09:58, Eric Edgar spammed:
>
> > You also have to concider that you were doing things incorrectly.  We
> > only have so much time in the day and have to go after the low hanging
> > fruit first per say.  While you are correct that something is wrong in
> > the tree the initial results did not indicate that.
>
> It took me a few iterations to understand how the tool is _supposed_ to be
> used.  At no place in the documentation does it clearly explain that the
> syntax of stage4/use: does not migrate to stage2/use:, for example, or that
> USE flags are NOT to be used anywhere but stage4, or that USE flags are not
> to go into the envscript.  There is one generic stage example, which has
> been subsequently changed to explain that cflags and such only apply on
> stage2, but no mention of use flags and so on.  I assumed that:

The templates for stage2 and stage3 don't list stage?/use.  Those are
the documentation for the specs, so it is mentioned in the documentation
pretty explicitly.

I think the problem was that you assumed.

> "The catalyst tool is intended to be used by those who wish to create their
> own customized versions of Gentoo Linux, or their own customized LiveCDs.
> Our goal is to make catalyst a powerful tool that's a pleasure to use, and
> to ensure that the code we write is maintainable and of high-quality."
>
> ..meant that the tool was intended to be used to create customized versions,
> what I didn't understand was the customization only applies in very limited
> ways to stage4...

No.  You can completely customize your lower stages, it just needs to be
done within the confines of what portage allows.  This means you need a
profile to make modifications.  Since stage4 allows you to do things
beyond the "system" target, it can be customized outside of a profile.
This is a limitation in portage which will not be worked around in
catalyst simply because I 100% agree with the limitation.

> So I have had to learn the hard way and have had to bug the hell out of you
> guys.  I would be glad to work on some of the documentation after wrapping
> my mind around it more completely :)

I am working on the documentation.  Right now, catalyst 1.x is the
stable tree.  Anything and everything in catalyst 2.x is still pretty
dynamic, as we are working in final features and bug fixes before
release.  I'll be working on the documentation a bit more as time
permits once 2.0 final is released.

> > put out a new gentoo cd.  There are only 2 main developers working on
> > catalyst at the moment, Myself and wolf31o2.
>
> Geez, no wonder the fuse is kinda short.  I had no idea it was only you two.

Until I brought rocket on as a developer, it was just myself maintaining
code that I had inherited, and the original developers were no longer
with the project.  Catalyst has come a long way since I took it over,
and it still has a long way to go.

> > from my standpoint I have seen you as quite firm in your convictions
> > that you were right.  I agree that the stage1 bug you posted is valid,
> >      but you could have also performed the few requested steps to follow
> >      wolf31o2's suggestions.
>
> After making a few boneheaded mistakes I did follow Chris's suggestions, and
> arrived at the same results.  I also understood that the actual problem is
> not caused by those mistakes.  Stage1 is the only stage that installs into
> a separate ROOT as far as I can tell.  Catalyst is not taking into account
> the order that portage.grab_stacked() and portage.stack_lists() is
> returning package lists in.  Glibc requires headers to be installed first.

No.  The problem was 100% a problem in the tree which has been fixed
(now).

> The solution is either to hack portage to always sort lists that involve
> the toolchain correctly, or to account for it in catalyst.  I have never

Portage handles all dependencies for a reason.  The toolchain on Gentoo
is not static.  What if you're using catalyst to build a Gentoo/FreeBSD
target?  What if you're using it to build a hardened uclibc MIPS target?

> coded in python, so I don't quite know how to implement the solution, but I
> do have a guess at what the solution is.  Push any toolchain package to the
> top (or bottom, whichever) of the buildpkgs array in their proper order,
> don't rely on portage.grab_stacked() or portage.stack_lists() to order it
> correctly because it does not gracefully take into account the install is
> happening in a different ROOT.  The DEPEND check in the glibc ebuild is
> looking for headers in the running system, not what is installed in ROOT,
> so it does not see it as a dependency that needs to be installed first.

See above.

> <rant>As an anal purist, I believe the toolchain should be installed first
> and in the proper order explicitly one package at a time instead of in a
> stacked list.  To do otherwise is heaping problems on yourself on down the
> road if you ask me.  You also avoid this weird dependency problem when
> installing into a ROOT.</rant>

We cannot enforce an order on an arbitrary set of packages.  That is
portage's job.  The problem that you were hitting was due to a bug in
the glibc ebuild which has been resolved.

> > Making a new profile is really very easy to do and with catalysts
> > portage_overlay features quite painless to do.  It works just like an
> > overlay on a host gentoo system.  I can understand that it might have
> > sounded difficult or time consuming to follow the recommendation but a
> > better course of action from you would have been how can I do this .. or
> > how do you guys do this.  If it sounds like it sucks for you maybe it
> > sucks for us too, but we just havent had the time to code a simpler way
> > yet.  We are not out to make anyones life difficult, remember we are
> > volunteers here with limited time.
>
> Can you put profiles in overlays?  Duh, that would be much easier.

Yes.  You can even inherit from the profiles in the live tree.  A
portage overlay is a complete overlay.  You can provide any files you
require.

> > you are also missing the fact that the profile stays consistent from one
> > stage to the next.  custom use flags in a spec file can and will very
> > easily be missed or dropped or typoed etc.  It really is a better design
> > choice to do it with profiles.
>
> Call this a feature request, or a suggestion.
>
> Allow cflags, chosts, cxxflags, FEATURES, and use flags in only ONE place -
> stage2.  Stage2 can generate the entire make.conf that will be used through
> any future stages, including the user specified USE flags. Stage2 runs the
> bootstrap.sh, which correctly throws out non-applicable USE flags, so it is
> not a problem, all the catalyst developers would have to do is count on
> bootstrap.sh to work correctly.

Most of this is done, except there are specific reasons for not doing as
you suggest.  For example, a stage1 can use differing FEATURES from a
stage2.  This is expected.  I will not ever allow customization of USE
flags in the lower stages (stages 1 through 3) as it breaks the static
concept of what each stage means.  The idea of a stage3 is it is a stage
that follows the profile's "system" target *to the letter* with *no
changes* in it.  Allowing for changes breaks this concept and will never
be accepted into catalyst.

> From that point forward every stage done by catalyst uses the USE flags
> generated in step 2 and never changes.  If a USE flag breaks the install,
> it would have broken during the install in stage4 anyway.  This would
> greatly simplify everything in my opinion.

No.  A USE flag can break the "system" target but not break beyond that
point.  This means a USE flag may be safe during the creation of a
stage4 tarball but not during a stage3 tarball.  Why?  Because during
the creation of stages 1 though 3 we only have a limited set of packages
and even those packages are compiled and configured in minimal ways.
Until we reach the "system" target we do not have a guaranteed working
complete system to work with.

> Stage1 would continue to be the proper place to sanitize the build to
> pre-selected cflags and use flags, etc...
>
> Am I crazy, or what?

You said it, not I.  ;]

Remember that we've been using catalyst for more than 2 years now to do
releases and have seen what can happen.  Most of the restrictions that
currently exist in catalyst were not there when we first started using
it and have been added because they allow breakages and are generally
not safe.  We didn't set out to arbitrarily restrict building.  It
happened over time due to limitations in portage as we discovered them.

> > You assume that its a simple thing to compile a tool chain in order.
> > But what you miss is the fact that upstream developers consistently
> > change how things are done.  We have coded catalyst the best way for us.
> > Using profiles allows us to leverage other developers time in the tree
> > to catch a majority of the issues for us.  They can help reorganize the
> > way things are built if necessary via a profile.  Otherwise we would be
> > constantly updating our spec files and never know exactly what was going
> > on.
>
> It _is_ a simple thing to bootstrap a toolchain in order.  The order has
> always been the exact same and probably always will be.  binutils, gcc,
> headers, glibc.  Repeat, using tools from previous step.  This is the

Except for uclibc... or libc on *BSD... or Mac OS... or Solaris...
remember that "Gentoo" is more than just Linux/glibc.

> _only_ way as far as I know to do it correctly, and to do it correctly
> every time, and to be absolutely certain it has been done correctly.  It
> has been this way for over 10 years if I recall correctly, it is linux
> system building 101.

As a tool, catalyst utilizes portage for determining all building.  If
the portage tree is broken, it is broken.  It isn't catalyst's place to
sanitize it.  It is the Gentoo developers' job as a whole to do so.  A
"Gentoo" toolchain can be any number of various combinations of things.
It doesn't have to stick to the Linux kernel/GNU userland model that
other distributions maintain.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead
x86 Architecture Team
Games - Developer
Gentoo Linux