Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Sam Pfeiffer]

Grobian, any input about this?

On Thu, Jan 16, 2020, 17:39 Sam Pfeiffer <sammypfeiffer@×××××.com> wrote:

> Hello Benda,
>
> To be more specific... To easily go thru it I made a gist with the latest
> version of *bootstrap-prefix.sh* and I'll be pointing to specific lines
> (this also helps me recap why I did this, as I remember using stage{1,2,3}
> didn't work for me when I first came into this):
>
> When you execute *bootstrap-prefix.sh* without arguments (other than the
> EPREFIX, *./bootstrap-prefix.sh /MY/EPREFIX*) it calls this line:
>
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
> Calling *bootstrap_interactive*
>
> If you give an extra argument (*./bootstrap-prefix.sh /MY/EPREFIX
> [noninteractive, stage{1,2,3}]*) it calls this line:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
> bootstrap_${TODO#non} || exit 1
>
> Which, on the noninteractive case, goes to the same place than no
> arguments (*bootstrap_interactive*). But in the stage{1,2,3} calls
> directly the *bootstrap_stageX* function.
>
> The thing is, noninteractive does some checks, finds resources and sets up
> environment variables like:
>
>    - Check for bad flags:
>    https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
>    - We find a gcc:
>    https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
>    - We get a cpu count and adjust -j for Make:
>    https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
>    - Check if we deal with multilib systems:
>    https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
>    - Check if we are boostrapping from a Gentoo system:
>    https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
>    - Set our EPREFIX:
>    https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
>
> Which is summarised at the end by setting the environment variables
> EPREFIX, CHOST, PATH, MAKEOPTS.
>
> And then finally calls *bootstrap_stage1_log*, which runs*
> bootstrap_stage1* itself.
>
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
>
> Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1...
> Calls *bootstrap_stage1()*:
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
>
> And as you can see it won't do any of the extra work done in
> *bootstrap_interactive()*.
>
> I'm not that versed in this script, so I may be missing something easy to
> overcome this (or I don't know enough about what is going on), but there is
> no separate function that does everything in* bootstrap_interactive*
> without calling *bootstrap_stage1*. And *bootstrap_stage1* does not do
> all that setup that *bootstrap_interactive* does. So that's how I came to
> my proposal.
>
> If there is some patch to enable that setup from *bootstrap_interactive*
> with* bootstrap_stage1*, that would be a good alternative, I guess. I
> still like the fact that with my approach you are just saying "bootstrap
> everything until this step" instead of needing to manually do
> *bootstrap_setup* (imaginary function I just named) then
> *bootstrap_stage1*, then *bootstrap_stage2*, then* bootstrap_stage3*, and
> then no argument for the last *emerge -e system*, to get to a specific
> bootstrap stage, but that's just commodity.
>
> Let me know if I'm wrong somewhere or you think of any better solution!
>
> On Thu, 16 Jan 2020 at 13:42, Sam Pfeiffer <sammypfeiffer@×××××.com>
> wrote:
>
>> Hi Benda,
>>
>> Thanks for your reply.
>>
>> On Thu, 16 Jan 2020 at 13:01, Benda Xu <heroxbd@g.o> wrote:
>>
>>> Hi Sam,
>>>
>>> Sam Pfeiffer <sammypfeiffer@×××××.com> writes:
>>>
>>> > To ease automated building and debugging of the bootstrap of Gentoo
>>> > Prefix I would like to propose a patch to add the option to stop the
>>> > non-interactive bootstrap on specific stages of the bootstrap.
>>> >
>>> > I'm currently doing this in my CI projects [1][2] and I'd love to have
>>> > this added upstream.
>>> >
>>> > I propose a patch like this:
>>> >https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
>>> >
>>> > Which is what I'm currently using. It just adds another optional
>>> >parameter to the commandline (a 3rd one) in non-interactive mode which
>>> >you tell it after which stage to stop (stage1, stage2, stage3).
>>>
>>> > I'm not the most versed person in bash, so maybe there is a more
>>> > conceptually beautiful way of doing this. My patch is just a proposal
>>> > (that has been tested and running).
>>> >
>>> > This patch is useful because it allows:
>>> > 1) Easier debugging by stages.
>>>
>>> > 2) Allows to split the bootstrap job in parts in CI environments with
>>> > time limits on jobs to run (e.g. Azure Pipelines time limit is 6h*,
>>> > and the bootstrap takes in between 5h40-6h40 depending on random
>>> > things like time to download things, and load on the machine... or
>>> > packages just taking longer to build).
>>> >
>>> > And this patch should not change anything to anyone else (AFAIK).
>>>
>>> Are you aware that in bootstrap-prefix.sh you can specify the stage you
>>> are after?  e.g.
>>>
>>>    https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
>>>
>>> The text above that could be outdated, but PATH exports and stage1~3 are
>>> correct.
>>>
>> I'm aware, but the behaviour (relating to environment variables and
>> choosing CPU numbers) differs from the
>> noninteractive argument. I'd like to keep the CI environment as close as
>> a user executing ./bootstrap-prefix.sh EPREFIX as possible.
>> That's why I'm proposing this addition. Another approach could be
>> modifying the current argument to do the same than the noninteractive one.
>> But that may break the workflow of someone, and I don't want that.
>>
>>
>>>
>>> > Thanks for your time and feedback!
>>> >
>>> > P.S.: I'm used to the GitHub/GitLab workflow of having a graphical
>>> > interface to do a Pull Request... I don't know how to do this with
>>> > https://gitweb.gentoo.org/repo/proj/prefix.git/ If there is no way,
>>> > aside of the goal of this email, I'd propose to have a mirror in
>>> > GitHub.
>>>
>>> Yes, that's a good to have for me, but not in high priority.
>>>
>>> > *If you are curious, Stage 1 takes 8min~, Stage 2 takes 29min~, Stage
>>> > 3 takes 2h30min~, emerge system takes 2h35min~ on Azure Pipelines,
>>> > approximately.
>>>
>>> Thank you so much for all your work!  That's extremely helpful to keep
>>> the quality of Prefix codebase.
>>>
>>> > [1] https://github.com/awesomebytes/gentoo_prefix_ci
>>> > [2] https://github.com/awesomebytes/gentoo_prefix_ci_32b
>>>
>>> Yours,
>>> Benda
>>>
>>>
>>
>> --
>>
>> *Sammy Pfeiffer*
>> PhD Candidate at The Magic Lab within UTS.
>>
>
>
> --
>
> *Sammy Pfeiffer*
> PhD Candidate at The Magic Lab within UTS.
>