Re: [gentoo-alt] Allow non-interactive bootstrap to be stopped at arbitrary stages

[Sam Pfeiffer]

Thanks a lot Fabian!

I've just adopted it and the first jobs are running with it :) This implies
I won't need to sync my bootstrap-prefix.sh with upstream by hand anymore,
that's great news!

On Sat, 13 Jun 2020 at 23:54, Fabian Groffen <grobian@g.o> wrote:

> I think pretty late, but this is committed in f6bbc6f470.
>
> Thanks,
> Fabian
>
> On 03-02-2020 14:54:26 +1100, Sam Pfeiffer wrote:
> > Hey Fabian,
> >
> > Thanks for chipping in. I do think it's useful and user-friendly to be
> able to
> > control the target of the interactive/noninteractive script.
> > I proposed in my first email in the thread something similar with the
> > $STOP_AFTER_STAGE variable, even though I was capturing it from the
> commandline,
> > and as I understand
> > you propose, it would probably be better to capture it from an
> environment
> > variable (it would actually be more in-line with other existing
> variables).
> >
> > Benda, I'm sorry for the noise I'm causing. I just want to improve the
> > experience for the next person that tries to use this amazing project by
> > taking advantage of already implemented stuff. And also just having this
> > discussion is good to have more information archived to google about
> Gentoo
> > Prefix and its bootstrap process. I think.
> >
> > On Sun, 2 Feb 2020 at 23:08, Fabian Groffen <grobian@g.o[1]>
> wrote:
> > > What I gather from this discussion is that the bootstrap script is
> > > complex, and grew more complexity over the years.
> > >
> > > Back in the early days, there was just documentation, one needed to
> > > follow.  Since this frequently needed an update, and people kept using
> > > older copies, at some point the bootstrap-prefix.sh script was
> > > introduced.
> > >
> > > This script followed the documentation to the letter.  It was an exact
> > > replica of the instructions in the bootstrap document.  This included
> > > the notions of 3 stages and a world recompile.
> > >
> > > Today, the purpose of the stages have blurred, and seem more like
> > > batches in which to do things.  Some of the initial intentions still
> > > stand out, e.g. that stage1 is there to provide a working portage
> > > environment with some minimal efforts.  Minimal got redefined as soon
> as
> > > we had to install python and more and more tools seemed necesary (sed,
> > > path, awk, etc.).
> > >
> > > Now, it seems you need to cut time and sort of bootstrap up-to a stage.
> > > Would it be an option to simply add a(nother) var that will control the
> > > interactive target in terms of how far it will run along.  Normal
> resume
> > > strategies it employs already would simply allow a next run to run
> along
> > > further.
> > >
> > > I think above can be done easily with a few lines of code.
> > >
> > > Let me know what you think.
> > >
> > > Thanks,
> > > Fabian
> > >
> > > On 23-01-2020 13:31:05 +1100, Sam Pfeiffer wrote:
> > > > Grobian, any input about this?
> > > >
> > > > On Thu, Jan 16, 2020, 17:39 Sam Pfeiffer <[1]sammypfeiffer@×××××.com
> [2]>
> > > wrote:
> > > >
> > > > > Hello Benda,
> > > >
> > > > > To be more specific... To easily go thru it I made a gist with the
> latest
> > > > > version of bootstrap-prefix.sh and I'll be pointing to specific
> lines (this
> > > > > also helps me recap why I did this, as I remember using
> stage{1,2,3} didn't
> > > > > work for me when I first came into this):
> > > >
> > > > > When you execute bootstrap-prefix.sh without arguments (other than
> the
> > > > > EPREFIX, ./bootstrap-prefix.sh /MY/EPREFIX) it calls this line:
> > > >
> > > > > [2]https://gist.github.com/awesomebytes/
> > > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033[3]
> > > >
> > > > > Calling bootstrap_interactive
> > > >
> > > > > If you give an extra argument (./bootstrap-prefix.sh /MY/EPREFIX
> > > > > [noninteractive, stage{1,2,3}]) it calls this line:
> > > > > [3]https://gist.github.com/awesomebytes/
> > > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082[4]
> > > >
> > > > > bootstrap_${TODO#non} || exit 1
> > > >
> > > > > Which, on the noninteractive case, goes to the same place than no
> arguments
> > > > > (bootstrap_interactive). But in the stage{1,2,3} calls directly the
> > > > > bootstrap_stageX function.
> > > >
> > > > > The thing is, noninteractive does some checks, finds resources and
> sets up
> > > > > environment variables like:
> > > >
> > > > >   * Check for bad flags:
> > > > >   [4]https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247[5]
> > > > >   * We find a gcc:
> > > > >   [5]https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433[6]
> > > > >   * We get a cpu count and adjust -j for Make:
> > > > >   [6]https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501[7]
> > > > >   * Check if we deal with multilib systems:
> > > > >   [7]https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576[8]
> > > > >   * Check if we are boostrapping from a Gentoo system:
> > > > >   [8]https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608[9]
> > > > >   * Set our EPREFIX:
> > > > > [9]https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681[10]
> > > >
> > > > > Which is summarised at the end by setting the environment
> variables EPREFIX,
> > > > > CHOST, PATH, MAKEOPTS.
> > > >
> > > > > And then finally calls bootstrap_stage1_log, which runs
> bootstrap_stage1
> > > > > itself.
> > > >
> > > > > [10]https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726[11]
> > > >
> > > > > Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1...
> > > >
> > > > > Calls bootstrap_stage1():
> > > > > [11]https://gist.github.com/awesomebytes/
> > > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355[12]
> > > >
> > > > > And as you can see it won't do any of the extra work done in
> > > > > bootstrap_interactive().
> > > >
> > > > > I'm not that versed in this script, so I may be missing something
> easy to
> > > > > overcome this (or I don't know enough about what is going on), but
> there is
> > > no
> > > > > separate function that does everything in bootstrap_interactive
> without
> > > > > calling bootstrap_stage1. And bootstrap_stage1 does not do all
> that setup
> > > that
> > > > > bootstrap_interactive does. So that's how I came to my proposal.
> > > >
> > > > > If there is some patch to enable that setup from
> bootstrap_interactive with
> > > > > bootstrap_stage1, that would be a good alternative, I guess. I
> still like
> > > the
> > > > > fact that with my approach you are just saying "bootstrap
> everything until
> > > > > this step" instead of needing to manually do bootstrap_setup
> (imaginary
> > > > > function I just named) then bootstrap_stage1, then
> bootstrap_stage2, then
> > > > > bootstrap_stage3, and then no argument for the last emerge -e
> system, to get
> > > > > to a specific bootstrap stage, but that's just commodity.
> > > >
> > > > > Let me know if I'm wrong somewhere or you think of any better
> solution!
> > > >
> > > > > On Thu, 16 Jan 2020 at 13:42, Sam Pfeiffer <[12]
> sammypfeiffer@×××××.com[13]>
> > > > > wrote:
> > > >
> > > > >> Hi Benda,
> > > >
> > > > >> Thanks for your reply.
> > > >
> > > > >> On Thu, 16 Jan 2020 at 13:01, Benda Xu <[13]heroxbd@g.o[14]>
> wrote:
> > > >
> > > > >>> Hi Sam,
> > > >
> > > > >>> Sam Pfeiffer <[14]sammypfeiffer@×××××.com[15]> writes:
> > > >
> > > > >>> > To ease automated building and debugging of the bootstrap of
> Gentoo
> > > > >>> > Prefix I would like to propose a patch to add the option to
> stop the
> > > > >>> > non-interactive bootstrap on specific stages of the bootstrap.
> > > > >>> >
> > > > >>> > I'm currently doing this in my CI projects [1][2] and I'd love
> to have
> > > > >>> > this added upstream.
> > > > >>> >
> > > > >>> > I propose a patch like this:
> > > > >>> >[15]
> https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> > > [16]
> > > > >>> >
> > > > >>> > Which is what I'm currently using. It just adds another
> optional
> > > > >>> >parameter to the commandline (a 3rd one) in non-interactive
> mode which
> > > > >>> >you tell it after which stage to stop (stage1, stage2, stage3).
> > > >
> > > > >>> > I'm not the most versed person in bash, so maybe there is a
> more
> > > > >>> > conceptually beautiful way of doing this. My patch is just a
> proposal
> > > > >>> > (that has been tested and running).
> > > > >>> >
> > > > >>> > This patch is useful because it allows:
> > > > >>> > 1) Easier debugging by stages.
> > > >
> > > > >>> > 2) Allows to split the bootstrap job in parts in CI
> environments with
> > > > >>> > time limits on jobs to run (e.g. Azure Pipelines time limit is
> 6h*,
> > > > >>> > and the bootstrap takes in between 5h40-6h40 depending on
> random
> > > > >>> > things like time to download things, and load on the
> machine... or
> > > > >>> > packages just taking longer to build).
> > > > >>> >
> > > > >>> > And this patch should not change anything to anyone else
> (AFAIK).
> > > >
> > > > >>> Are you aware that in bootstrap-prefix.sh you can specify the
> stage you
> > > > >>> are after?  e.g.
> > > >
> > > > >>>    [16]https://wiki.gentoo.org/wiki/Project:Prefix/
> > > Manual_Bootstrap#Stage_1[17]
> > > >
> > > > >>> The text above that could be outdated, but PATH exports and
> stage1~3 are
> > > > >>> correct.
> > > >
> > > > >> I'm aware, but the behaviour (relating to environment variables
> and
> > > choosing
> > > > >> CPU numbers) differs from the
> > > >
> > > > >> noninteractive argument. I'd like to keep the CI environment as
> close as a
> > > > >> user executing ./bootstrap-prefix.sh EPREFIX as possible.
> > > >
> > > > >> That's why I'm proposing this addition. Another approach could be
> modifying
> > > > >> the current argument to do the same than the noninteractive one.
> > > >
> > > > >> But that may break the workflow of someone, and I don't want that.
> > > >
> > > > >>
> > > >
> > > > >>> > Thanks for your time and feedback!
> > > > >>> >
> > > > >>> > P.S.: I'm used to the GitHub/GitLab workflow of having a
> graphical
> > > > >>> > interface to do a Pull Request... I don't know how to do this
> with
> > > > >>> > [17]https://gitweb.gentoo.org/repo/proj/prefix.git/[18] If
> there is no
> > > way,
> > > > >>> > aside of the goal of this email, I'd propose to have a mirror
> in
> > > > >>> > GitHub.
> > > >
> > > > >>> Yes, that's a good to have for me, but not in high priority.
> > > >
> > > > >>> > *If you are curious, Stage 1 takes 8min~, Stage 2 takes
> 29min~, Stage
> > > > >>> > 3 takes 2h30min~, emerge system takes 2h35min~ on Azure
> Pipelines,
> > > > >>> > approximately.
> > > >
> > > > >>> Thank you so much for all your work!  That's extremely helpful
> to keep
> > > > >>> the quality of Prefix codebase.
> > > >
> > > > >>> > [1] [18]https://github.com/awesomebytes/gentoo_prefix_ci[19]
> > > > >>> > [2] [19]
> https://github.com/awesomebytes/gentoo_prefix_ci_32b[20]
> > > >
> > > > >>> Yours,
> > > > >>> Benda
> > > >
> > > > >> --
> > > >
> > > > >> Sammy Pfeiffer
> > > > >> PhD Candidate at The Magic Lab within UTS.
> > > >
> > > > > --
> > > >
> > > > > Sammy Pfeiffer
> > > > > PhD Candidate at The Magic Lab within UTS.
> > > >
> > > >
> > > >
> > > >  References:
> > > >    1. mailto:sammypfeiffer@×××××.com[21]
> > > >    2. https://gist.github.com/awesomebytes/
> > > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033[22]
> > > >    3. https://gist.github.com/awesomebytes/
> > > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082[23]
> > > >    4. https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247[24]
> > > >    5. https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433[25]
> > > >    6. https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501[26]
> > > >    7. https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576[27]
> > > >    8. https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608[28]
> > > >    9. https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681[29]
> > > >   10. https://gist.github.com/awesomebytes/
> > >
> ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726[30]
> > > >   11. https://gist.github.com/awesomebytes/
> > > ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355[31]
> > > >   12. mailto:sammypfeiffer@×××××.com[32]
> > > >   13. mailto:heroxbd@g.o[33]
> > > >   14. mailto:sammypfeiffer@×××××.com[34]
> > > >   15.
> https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> > > [35]
> > > >   16.
> https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1[36]
> > > >   17. https://gitweb.gentoo.org/repo/proj/prefix.git/[37]
> > > >   18. https://github.com/awesomebytes/gentoo_prefix_ci[38]
> > > >   19. https://github.com/awesomebytes/gentoo_prefix_ci_32b[39]
> > > >
> > > > read_char: errno==EILSEQ; invalid byte sequence for UTF-8:
> > > --
> > > Fabian Groffen
> > > Gentoo on a different level
> >
> >
> > --
> >
> > Sammy Pfeiffer
> > PhD Candidate at The Magic Lab within UTS.
> >
> >
> > References
> >    1. mailto:grobian@g.o
> >    2. mailto:sammypfeiffer@×××××.com
> >    3. https://gist.github.com/awesomebytes/
> >       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
> >    4. https://gist.github.com/awesomebytes/
> >       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
> >    5. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
> >    6. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
> >    7. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
> >    8. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
> >    9. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
> >   10. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
> >   11. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
> >   12. https://gist.github.com/awesomebytes/
> >       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
> >   13. mailto:sammypfeiffer@×××××.com
> >   14. mailto:heroxbd@g.o
> >   15. mailto:sammypfeiffer@×××××.com
> >   16.
> https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> >   17.
> https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
> >   18. https://gitweb.gentoo.org/repo/proj/prefix.git/
> >   19. https://github.com/awesomebytes/gentoo_prefix_ci
> >   20. https://github.com/awesomebytes/gentoo_prefix_ci_32b
> >   21. mailto:sammypfeiffer@×××××.com
> >   22. https://gist.github.com/awesomebytes/
> >       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033
> >   23. https://gist.github.com/awesomebytes/
> >       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082
> >   24. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247
> >   25. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433
> >   26. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501
> >   27. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576
> >   28. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608
> >   29. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681
> >   30. https://gist.github.com/awesomebytes/
> >
>  ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726
> >   31. https://gist.github.com/awesomebytes/
> >       ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355
> >   32. mailto:sammypfeiffer@×××××.com
> >   33. mailto:heroxbd@g.o
> >   34. mailto:sammypfeiffer@×××××.com
> >   35.
> https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f
> >   36.
> https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1
> >   37. https://gitweb.gentoo.org/repo/proj/prefix.git/
> >   38. https://github.com/awesomebytes/gentoo_prefix_ci
> >   39. https://github.com/awesomebytes/gentoo_prefix_ci_32b
>
> --
> Fabian Groffen
> Gentoo on a different level
>


-- 

*Sammy Pfeiffer*
PhD Candidate at The Magic Lab within UTS.