Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Fabian Groffen]

I think pretty late, but this is committed in f6bbc6f470.

Thanks,
Fabian

On 03-02-2020 14:54:26 +1100, Sam Pfeiffer wrote:
> Hey Fabian,
>
> Thanks for chipping in. I do think it's useful and user-friendly to be able to
> control the target of the interactive/noninteractive script.
> I proposed in my first email in the thread something similar with the
> $STOP_AFTER_STAGE variable, even though I was capturing it from the commandline,
> and as I understand
> you propose, it would probably be better to capture it from an environment
> variable (it would actually be more in-line with other existing variables).
>
> Benda, I'm sorry for the noise I'm causing. I just want to improve the
> experience for the next person that tries to use this amazing project by
> taking advantage of already implemented stuff. And also just having this
> discussion is good to have more information archived to google about Gentoo
> Prefix and its bootstrap process. I think.
>
> On Sun, 2 Feb 2020 at 23:08, Fabian Groffen <grobian@g.o[1]> wrote:
> > What I gather from this discussion is that the bootstrap script is
> > complex, and grew more complexity over the years.
> >
> > Back in the early days, there was just documentation, one needed to
> > follow.  Since this frequently needed an update, and people kept using
> > older copies, at some point the bootstrap-prefix.sh script was
> > introduced.
> >
> > This script followed the documentation to the letter.  It was an exact
> > replica of the instructions in the bootstrap document.  This included
> > the notions of 3 stages and a world recompile.
> >
> > Today, the purpose of the stages have blurred, and seem more like
> > batches in which to do things.  Some of the initial intentions still
> > stand out, e.g. that stage1 is there to provide a working portage
> > environment with some minimal efforts.  Minimal got redefined as soon as
> > we had to install python and more and more tools seemed necesary (sed,
> > path, awk, etc.).
> >
> > Now, it seems you need to cut time and sort of bootstrap up-to a stage.
> > Would it be an option to simply add a(nother) var that will control the
> > interactive target in terms of how far it will run along.  Normal resume
> > strategies it employs already would simply allow a next run to run along
> > further.
> >
> > I think above can be done easily with a few lines of code.
> >
> > Let me know what you think.
> >
> > Thanks,
> > Fabian
> >
> > On 23-01-2020 13:31:05 +1100, Sam Pfeiffer wrote:
> > > Grobian, any input about this?
> > >
> > > On Thu, Jan 16, 2020, 17:39 Sam Pfeiffer <[1]sammypfeiffer@×××××.com[2]>
> > wrote:
> > >
> > > > Hello Benda,
> > >
> > > > To be more specific... To easily go thru it I made a gist with the latest
> > > > version of bootstrap-prefix.sh and I'll be pointing to specific lines (this
> > > > also helps me recap why I did this, as I remember using stage{1,2,3} didn't
> > > > work for me when I first came into this):
> > >
> > > > When you execute bootstrap-prefix.sh without arguments (other than the
> > > > EPREFIX, ./bootstrap-prefix.sh /MY/EPREFIX) it calls this line:
> > >
> > > > [2]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033[3]
> > >
> > > > Calling bootstrap_interactive
> > >
> > > > If you give an extra argument (./bootstrap-prefix.sh /MY/EPREFIX
> > > > [noninteractive, stage{1,2,3}]) it calls this line:
> > > > [3]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082[4]
> > >
> > > > bootstrap_${TODO#non} || exit 1
> > >
> > > > Which, on the noninteractive case, goes to the same place than no arguments
> > > > (bootstrap_interactive). But in the stage{1,2,3} calls directly the
> > > > bootstrap_stageX function.
> > >
> > > > The thing is, noninteractive does some checks, finds resources and sets up
> > > > environment variables like:
> > >
> > > >   * Check for bad flags:
> > > >   [4]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247[5]
> > > >   * We find a gcc:
> > > >   [5]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433[6]
> > > >   * We get a cpu count and adjust -j for Make:
> > > >   [6]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501[7]
> > > >   * Check if we deal with multilib systems:
> > > >   [7]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576[8]
> > > >   * Check if we are boostrapping from a Gentoo system:
> > > >   [8]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608[9]
> > > >   * Set our EPREFIX:
> > > > [9]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681[10]
> > >
> > > > Which is summarised at the end by setting the environment variables EPREFIX,
> > > > CHOST, PATH, MAKEOPTS.
> > >
> > > > And then finally calls bootstrap_stage1_log, which runs bootstrap_stage1
> > > > itself.
> > >
> > > > [10]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726[11]
> > >
> > > > Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1...
> > >
> > > > Calls bootstrap_stage1():
> > > > [11]https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355[12]
> > >
> > > > And as you can see it won't do any of the extra work done in
> > > > bootstrap_interactive().
> > >
> > > > I'm not that versed in this script, so I may be missing something easy to
> > > > overcome this (or I don't know enough about what is going on), but there is
> > no
> > > > separate function that does everything in bootstrap_interactive without
> > > > calling bootstrap_stage1. And bootstrap_stage1 does not do all that setup
> > that
> > > > bootstrap_interactive does. So that's how I came to my proposal.
> > >
> > > > If there is some patch to enable that setup from bootstrap_interactive with
> > > > bootstrap_stage1, that would be a good alternative, I guess. I still like
> > the
> > > > fact that with my approach you are just saying "bootstrap everything until
> > > > this step" instead of needing to manually do bootstrap_setup (imaginary
> > > > function I just named) then bootstrap_stage1, then bootstrap_stage2, then
> > > > bootstrap_stage3, and then no argument for the last emerge -e system, to get
> > > > to a specific bootstrap stage, but that's just commodity.
> > >
> > > > Let me know if I'm wrong somewhere or you think of any better solution!
> > >
> > > > On Thu, 16 Jan 2020 at 13:42, Sam Pfeiffer <[12]sammypfeiffer@×××××.com[13]>
> > > > wrote:
> > >
> > > >> Hi Benda,
> > >
> > > >> Thanks for your reply.
> > >
> > > >> On Thu, 16 Jan 2020 at 13:01, Benda Xu <[13]heroxbd@g.o[14]> wrote:
> > >
> > > >>> Hi Sam,
> > >
> > > >>> Sam Pfeiffer <[14]sammypfeiffer@×××××.com[15]> writes:
> > >
> > > >>> > To ease automated building and debugging of the bootstrap of Gentoo
> > > >>> > Prefix I would like to propose a patch to add the option to stop the
> > > >>> > non-interactive bootstrap on specific stages of the bootstrap.
> > > >>> >
> > > >>> > I'm currently doing this in my CI projects [1][2] and I'd love to have
> > > >>> > this added upstream.
> > > >>> >
> > > >>> > I propose a patch like this:
> > > >>> >[15]https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> > [16]
> > > >>> >
> > > >>> > Which is what I'm currently using. It just adds another optional
> > > >>> >parameter to the commandline (a 3rd one) in non-interactive mode which
> > > >>> >you tell it after which stage to stop (stage1, stage2, stage3).
> > >
> > > >>> > I'm not the most versed person in bash, so maybe there is a more
> > > >>> > conceptually beautiful way of doing this. My patch is just a proposal
> > > >>> > (that has been tested and running).
> > > >>> >
> > > >>> > This patch is useful because it allows:
> > > >>> > 1) Easier debugging by stages.
> > >
> > > >>> > 2) Allows to split the bootstrap job in parts in CI environments with
> > > >>> > time limits on jobs to run (e.g. Azure Pipelines time limit is 6h*,
> > > >>> > and the bootstrap takes in between 5h40-6h40 depending on random
> > > >>> > things like time to download things, and load on the machine... or
> > > >>> > packages just taking longer to build).
> > > >>> >
> > > >>> > And this patch should not change anything to anyone else (AFAIK).
> > >
> > > >>> Are you aware that in bootstrap-prefix.sh you can specify the stage you
> > > >>> are after?  e.g.
> > >
> > > >>>    [16]https://wiki.gentoo.org/wiki/Project:Prefix/
> > Manual_Bootstrap#Stage_1[17]
> > >
> > > >>> The text above that could be outdated, but PATH exports and stage1~3 are
> > > >>> correct.
> > >
> > > >> I'm aware, but the behaviour (relating to environment variables and
> > choosing
> > > >> CPU numbers) differs from the
> > >
> > > >> noninteractive argument. I'd like to keep the CI environment as close as a
> > > >> user executing ./bootstrap-prefix.sh EPREFIX as possible.
> > >
> > > >> That's why I'm proposing this addition. Another approach could be modifying
> > > >> the current argument to do the same than the noninteractive one.
> > >
> > > >> But that may break the workflow of someone, and I don't want that.
> > >
> > > >>  
> > >
> > > >>> > Thanks for your time and feedback!
> > > >>> >
> > > >>> > P.S.: I'm used to the GitHub/GitLab workflow of having a graphical
> > > >>> > interface to do a Pull Request... I don't know how to do this with
> > > >>> > [17]https://gitweb.gentoo.org/repo/proj/prefix.git/[18] If there is no
> > way,
> > > >>> > aside of the goal of this email, I'd propose to have a mirror in
> > > >>> > GitHub.
> > >
> > > >>> Yes, that's a good to have for me, but not in high priority.
> > >
> > > >>> > *If you are curious, Stage 1 takes 8min~, Stage 2 takes 29min~, Stage
> > > >>> > 3 takes 2h30min~, emerge system takes 2h35min~ on Azure Pipelines,
> > > >>> > approximately.
> > >
> > > >>> Thank you so much for all your work!  That's extremely helpful to keep
> > > >>> the quality of Prefix codebase.
> > >
> > > >>> > [1] [18]https://github.com/awesomebytes/gentoo_prefix_ci[19]
> > > >>> > [2] [19]https://github.com/awesomebytes/gentoo_prefix_ci_32b[20]
> > >
> > > >>> Yours,
> > > >>> Benda
> > >
> > > >> --
> > >
> > > >> Sammy Pfeiffer
> > > >> PhD Candidate at The Magic Lab within UTS.
> > >
> > > > --
> > >
> > > > Sammy Pfeiffer
> > > > PhD Candidate at The Magic Lab within UTS.
> > >
> > >
> > >
> > >  References:
> > >    1. mailto:sammypfeiffer@×××××.com[21]
> > >    2. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033[22]
> > >    3. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082[23]
> > >    4. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247[24]
> > >    5. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433[25]
> > >    6. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501[26]
> > >    7. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576[27]
> > >    8. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608[28]
> > >    9. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681[29]
> > >   10. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726[30]
> > >   11. https://gist.github.com/awesomebytes/
> > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355[31]
> > >   12. mailto:sammypfeiffer@×××××.com[32]
> > >   13. mailto:heroxbd@g.o[33]
> > >   14. mailto:sammypfeiffer@×××××.com[34]
> > >   15. https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> > [35]
> > >   16. https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1[36]
> > >   17. https://gitweb.gentoo.org/repo/proj/prefix.git/[37]
> > >   18. https://github.com/awesomebytes/gentoo_prefix_ci[38]
> > >   19. https://github.com/awesomebytes/gentoo_prefix_ci_32b[39]
> > >
> > > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> > --
> > Fabian Groffen
> > Gentoo on a different level
>
>
> --
>
> Sammy Pfeiffer
> PhD Candidate at The Magic Lab within UTS.
>
>
> References
>    1. mailto:grobian@g.o
>    2. mailto:sammypfeiffer@×××××.com
>    3. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
>    4. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
>    5. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
>    6. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
>    7. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
>    8. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
>    9. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
>   10. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
>   11. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
>   12. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
>   13. mailto:sammypfeiffer@×××××.com
>   14. mailto:heroxbd@g.o
>   15. mailto:sammypfeiffer@×××××.com
>   16. https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
>   17. https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
>   18. https://gitweb.gentoo.org/repo/proj/prefix.git/
>   19. https://github.com/awesomebytes/gentoo_prefix_ci
>   20. https://github.com/awesomebytes/gentoo_prefix_ci_32b
>   21. mailto:sammypfeiffer@×××××.com
>   22. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
>   23. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
>   24. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
>   25. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
>   26. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
>   27. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
>   28. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
>   29. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
>   30. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
>   31. https://gist.github.com/awesomebytes/
>       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
>   32. mailto:sammypfeiffer@×××××.com
>   33. mailto:heroxbd@g.o
>   34. mailto:sammypfeiffer@×××××.com
>   35. https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
>   36. https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
>   37. https://gitweb.gentoo.org/repo/proj/prefix.git/
>   38. https://github.com/awesomebytes/gentoo_prefix_ci
>   39. https://github.com/awesomebytes/gentoo_prefix_ci_32b

-- 
Fabian Groffen
Gentoo on a different level