Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Sam Pfeiffer]

Hello Benda,

To be more specific... To easily go thru it I made a gist with the latest
version of *bootstrap-prefix.sh* and I'll be pointing to specific lines
(this also helps me recap why I did this, as I remember using stage{1,2,3}
didn't work for me when I first came into this):

When you execute *bootstrap-prefix.sh* without arguments (other than the
EPREFIX, *./bootstrap-prefix.sh /MY/EPREFIX*) it calls this line:
https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
Calling *bootstrap_interactive*

If you give an extra argument (*./bootstrap-prefix.sh /MY/EPREFIX
[noninteractive, stage{1,2,3}]*) it calls this line:
https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
bootstrap_${TODO#non} || exit 1

Which, on the noninteractive case, goes to the same place than no arguments
(*bootstrap_interactive*). But in the stage{1,2,3} calls directly the
*bootstrap_stageX* function.

The thing is, noninteractive does some checks, finds resources and sets up
environment variables like:

   - Check for bad flags:
   https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
   - We find a gcc:
   https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
   - We get a cpu count and adjust -j for Make:
   https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
   - Check if we deal with multilib systems:
   https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
   - Check if we are boostrapping from a Gentoo system:
   https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
   - Set our EPREFIX:
   https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681

Which is summarised at the end by setting the environment variables
EPREFIX, CHOST, PATH, MAKEOPTS.

And then finally calls *bootstrap_stage1_log*, which runs* bootstrap_stage1*
itself.
https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726

Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1...
Calls *bootstrap_stage1()*:
https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355

And as you can see it won't do any of the extra work done in
*bootstrap_interactive()*.

I'm not that versed in this script, so I may be missing something easy to
overcome this (or I don't know enough about what is going on), but there is
no separate function that does everything in* bootstrap_interactive*
without calling *bootstrap_stage1*. And *bootstrap_stage1* does not do all
that setup that *bootstrap_interactive* does. So that's how I came to my
proposal.

If there is some patch to enable that setup from *bootstrap_interactive*
with* bootstrap_stage1*, that would be a good alternative, I guess. I still
like the fact that with my approach you are just saying "bootstrap
everything until this step" instead of needing to manually do
*bootstrap_setup* (imaginary function I just named) then *bootstrap_stage1*,
then *bootstrap_stage2*, then* bootstrap_stage3*, and then no argument for
the last *emerge -e system*, to get to a specific bootstrap stage, but
that's just commodity.

Let me know if I'm wrong somewhere or you think of any better solution!

On Thu, 16 Jan 2020 at 13:42, Sam Pfeiffer <sammypfeiffer@×××××.com> wrote:

> Hi Benda,
>
> Thanks for your reply.
>
> On Thu, 16 Jan 2020 at 13:01, Benda Xu <heroxbd@g.o> wrote:
>
>> Hi Sam,
>>
>> Sam Pfeiffer <sammypfeiffer@×××××.com> writes:
>>
>> > To ease automated building and debugging of the bootstrap of Gentoo
>> > Prefix I would like to propose a patch to add the option to stop the
>> > non-interactive bootstrap on specific stages of the bootstrap.
>> >
>> > I'm currently doing this in my CI projects [1][2] and I'd love to have
>> > this added upstream.
>> >
>> > I propose a patch like this:
>> >https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
>> >
>> > Which is what I'm currently using. It just adds another optional
>> >parameter to the commandline (a 3rd one) in non-interactive mode which
>> >you tell it after which stage to stop (stage1, stage2, stage3).
>>
>> > I'm not the most versed person in bash, so maybe there is a more
>> > conceptually beautiful way of doing this. My patch is just a proposal
>> > (that has been tested and running).
>> >
>> > This patch is useful because it allows:
>> > 1) Easier debugging by stages.
>>
>> > 2) Allows to split the bootstrap job in parts in CI environments with
>> > time limits on jobs to run (e.g. Azure Pipelines time limit is 6h*,
>> > and the bootstrap takes in between 5h40-6h40 depending on random
>> > things like time to download things, and load on the machine... or
>> > packages just taking longer to build).
>> >
>> > And this patch should not change anything to anyone else (AFAIK).
>>
>> Are you aware that in bootstrap-prefix.sh you can specify the stage you
>> are after?  e.g.
>>
>>    https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
>>
>> The text above that could be outdated, but PATH exports and stage1~3 are
>> correct.
>>
> I'm aware, but the behaviour (relating to environment variables and
> choosing CPU numbers) differs from the
> noninteractive argument. I'd like to keep the CI environment as close as a
> user executing ./bootstrap-prefix.sh EPREFIX as possible.
> That's why I'm proposing this addition. Another approach could be
> modifying the current argument to do the same than the noninteractive one.
> But that may break the workflow of someone, and I don't want that.
>
>
>>
>> > Thanks for your time and feedback!
>> >
>> > P.S.: I'm used to the GitHub/GitLab workflow of having a graphical
>> > interface to do a Pull Request... I don't know how to do this with
>> > https://gitweb.gentoo.org/repo/proj/prefix.git/ If there is no way,
>> > aside of the goal of this email, I'd propose to have a mirror in
>> > GitHub.
>>
>> Yes, that's a good to have for me, but not in high priority.
>>
>> > *If you are curious, Stage 1 takes 8min~, Stage 2 takes 29min~, Stage
>> > 3 takes 2h30min~, emerge system takes 2h35min~ on Azure Pipelines,
>> > approximately.
>>
>> Thank you so much for all your work!  That's extremely helpful to keep
>> the quality of Prefix codebase.
>>
>> > [1] https://github.com/awesomebytes/gentoo_prefix_ci
>> > [2] https://github.com/awesomebytes/gentoo_prefix_ci_32b
>>
>> Yours,
>> Benda
>>
>>
>
> --
>
> *Sammy Pfeiffer*
> PhD Candidate at The Magic Lab within UTS.
>


-- 

*Sammy Pfeiffer*
PhD Candidate at The Magic Lab within UTS.