1 |
What I gather from this discussion is that the bootstrap script is |
2 |
complex, and grew more complexity over the years. |
3 |
|
4 |
Back in the early days, there was just documentation, one needed to |
5 |
follow. Since this frequently needed an update, and people kept using |
6 |
older copies, at some point the bootstrap-prefix.sh script was |
7 |
introduced. |
8 |
|
9 |
This script followed the documentation to the letter. It was an exact |
10 |
replica of the instructions in the bootstrap document. This included |
11 |
the notions of 3 stages and a world recompile. |
12 |
|
13 |
Today, the purpose of the stages have blurred, and seem more like |
14 |
batches in which to do things. Some of the initial intentions still |
15 |
stand out, e.g. that stage1 is there to provide a working portage |
16 |
environment with some minimal efforts. Minimal got redefined as soon as |
17 |
we had to install python and more and more tools seemed necesary (sed, |
18 |
path, awk, etc.). |
19 |
|
20 |
Now, it seems you need to cut time and sort of bootstrap up-to a stage. |
21 |
Would it be an option to simply add a(nother) var that will control the |
22 |
interactive target in terms of how far it will run along. Normal resume |
23 |
strategies it employs already would simply allow a next run to run along |
24 |
further. |
25 |
|
26 |
I think above can be done easily with a few lines of code. |
27 |
|
28 |
Let me know what you think. |
29 |
|
30 |
Thanks, |
31 |
Fabian |
32 |
|
33 |
On 23-01-2020 13:31:05 +1100, Sam Pfeiffer wrote: |
34 |
> Grobian, any input about this? |
35 |
> |
36 |
> On Thu, Jan 16, 2020, 17:39 Sam Pfeiffer <[1]sammypfeiffer@×××××.com> wrote: |
37 |
> |
38 |
> > Hello Benda, |
39 |
> |
40 |
> > To be more specific... To easily go thru it I made a gist with the latest |
41 |
> > version of bootstrap-prefix.sh and I'll be pointing to specific lines (this |
42 |
> > also helps me recap why I did this, as I remember using stage{1,2,3} didn't |
43 |
> > work for me when I first came into this): |
44 |
> |
45 |
> > When you execute bootstrap-prefix.sh without arguments (other than the |
46 |
> > EPREFIX, ./bootstrap-prefix.sh /MY/EPREFIX) it calls this line: |
47 |
> |
48 |
> > [2]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033 |
49 |
> |
50 |
> > Calling bootstrap_interactive |
51 |
> |
52 |
> > If you give an extra argument (./bootstrap-prefix.sh /MY/EPREFIX |
53 |
> > [noninteractive, stage{1,2,3}]) it calls this line: |
54 |
> > [3]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082 |
55 |
> |
56 |
> > bootstrap_${TODO#non} || exit 1 |
57 |
> |
58 |
> > Which, on the noninteractive case, goes to the same place than no arguments |
59 |
> > (bootstrap_interactive). But in the stage{1,2,3} calls directly the |
60 |
> > bootstrap_stageX function. |
61 |
> |
62 |
> > The thing is, noninteractive does some checks, finds resources and sets up |
63 |
> > environment variables like: |
64 |
> |
65 |
> > * Check for bad flags: |
66 |
> > [4]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247 |
67 |
> > * We find a gcc: |
68 |
> > [5]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433 |
69 |
> > * We get a cpu count and adjust -j for Make: |
70 |
> > [6]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501 |
71 |
> > * Check if we deal with multilib systems: |
72 |
> > [7]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576 |
73 |
> > * Check if we are boostrapping from a Gentoo system: |
74 |
> > [8]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608 |
75 |
> > * Set our EPREFIX: |
76 |
> > [9]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681 |
77 |
> |
78 |
> > Which is summarised at the end by setting the environment variables EPREFIX, |
79 |
> > CHOST, PATH, MAKEOPTS. |
80 |
> |
81 |
> > And then finally calls bootstrap_stage1_log, which runs bootstrap_stage1 |
82 |
> > itself. |
83 |
> |
84 |
> > [10]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726 |
85 |
> |
86 |
> > Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1... |
87 |
> |
88 |
> > Calls bootstrap_stage1(): |
89 |
> > [11]https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355 |
90 |
> |
91 |
> > And as you can see it won't do any of the extra work done in |
92 |
> > bootstrap_interactive(). |
93 |
> |
94 |
> > I'm not that versed in this script, so I may be missing something easy to |
95 |
> > overcome this (or I don't know enough about what is going on), but there is no |
96 |
> > separate function that does everything in bootstrap_interactive without |
97 |
> > calling bootstrap_stage1. And bootstrap_stage1 does not do all that setup that |
98 |
> > bootstrap_interactive does. So that's how I came to my proposal. |
99 |
> |
100 |
> > If there is some patch to enable that setup from bootstrap_interactive with |
101 |
> > bootstrap_stage1, that would be a good alternative, I guess. I still like the |
102 |
> > fact that with my approach you are just saying "bootstrap everything until |
103 |
> > this step" instead of needing to manually do bootstrap_setup (imaginary |
104 |
> > function I just named) then bootstrap_stage1, then bootstrap_stage2, then |
105 |
> > bootstrap_stage3, and then no argument for the last emerge -e system, to get |
106 |
> > to a specific bootstrap stage, but that's just commodity. |
107 |
> |
108 |
> > Let me know if I'm wrong somewhere or you think of any better solution! |
109 |
> |
110 |
> > On Thu, 16 Jan 2020 at 13:42, Sam Pfeiffer <[12]sammypfeiffer@×××××.com> |
111 |
> > wrote: |
112 |
> |
113 |
> >> Hi Benda, |
114 |
> |
115 |
> >> Thanks for your reply. |
116 |
> |
117 |
> >> On Thu, 16 Jan 2020 at 13:01, Benda Xu <[13]heroxbd@g.o> wrote: |
118 |
> |
119 |
> >>> Hi Sam, |
120 |
> |
121 |
> >>> Sam Pfeiffer <[14]sammypfeiffer@×××××.com> writes: |
122 |
> |
123 |
> >>> > To ease automated building and debugging of the bootstrap of Gentoo |
124 |
> >>> > Prefix I would like to propose a patch to add the option to stop the |
125 |
> >>> > non-interactive bootstrap on specific stages of the bootstrap. |
126 |
> >>> > |
127 |
> >>> > I'm currently doing this in my CI projects [1][2] and I'd love to have |
128 |
> >>> > this added upstream. |
129 |
> >>> > |
130 |
> >>> > I propose a patch like this: |
131 |
> >>> >[15]https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f |
132 |
> >>> > |
133 |
> >>> > Which is what I'm currently using. It just adds another optional |
134 |
> >>> >parameter to the commandline (a 3rd one) in non-interactive mode which |
135 |
> >>> >you tell it after which stage to stop (stage1, stage2, stage3). |
136 |
> |
137 |
> >>> > I'm not the most versed person in bash, so maybe there is a more |
138 |
> >>> > conceptually beautiful way of doing this. My patch is just a proposal |
139 |
> >>> > (that has been tested and running). |
140 |
> >>> > |
141 |
> >>> > This patch is useful because it allows: |
142 |
> >>> > 1) Easier debugging by stages. |
143 |
> |
144 |
> >>> > 2) Allows to split the bootstrap job in parts in CI environments with |
145 |
> >>> > time limits on jobs to run (e.g. Azure Pipelines time limit is 6h*, |
146 |
> >>> > and the bootstrap takes in between 5h40-6h40 depending on random |
147 |
> >>> > things like time to download things, and load on the machine... or |
148 |
> >>> > packages just taking longer to build). |
149 |
> >>> > |
150 |
> >>> > And this patch should not change anything to anyone else (AFAIK). |
151 |
> |
152 |
> >>> Are you aware that in bootstrap-prefix.sh you can specify the stage you |
153 |
> >>> are after? e.g. |
154 |
> |
155 |
> >>> [16]https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1 |
156 |
> |
157 |
> >>> The text above that could be outdated, but PATH exports and stage1~3 are |
158 |
> >>> correct. |
159 |
> |
160 |
> >> I'm aware, but the behaviour (relating to environment variables and choosing |
161 |
> >> CPU numbers) differs from the |
162 |
> |
163 |
> >> noninteractive argument. I'd like to keep the CI environment as close as a |
164 |
> >> user executing ./bootstrap-prefix.sh EPREFIX as possible. |
165 |
> |
166 |
> >> That's why I'm proposing this addition. Another approach could be modifying |
167 |
> >> the current argument to do the same than the noninteractive one. |
168 |
> |
169 |
> >> But that may break the workflow of someone, and I don't want that. |
170 |
> |
171 |
> >> |
172 |
> |
173 |
> >>> > Thanks for your time and feedback! |
174 |
> >>> > |
175 |
> >>> > P.S.: I'm used to the GitHub/GitLab workflow of having a graphical |
176 |
> >>> > interface to do a Pull Request... I don't know how to do this with |
177 |
> >>> > [17]https://gitweb.gentoo.org/repo/proj/prefix.git/ If there is no way, |
178 |
> >>> > aside of the goal of this email, I'd propose to have a mirror in |
179 |
> >>> > GitHub. |
180 |
> |
181 |
> >>> Yes, that's a good to have for me, but not in high priority. |
182 |
> |
183 |
> >>> > *If you are curious, Stage 1 takes 8min~, Stage 2 takes 29min~, Stage |
184 |
> >>> > 3 takes 2h30min~, emerge system takes 2h35min~ on Azure Pipelines, |
185 |
> >>> > approximately. |
186 |
> |
187 |
> >>> Thank you so much for all your work! That's extremely helpful to keep |
188 |
> >>> the quality of Prefix codebase. |
189 |
> |
190 |
> >>> > [1] [18]https://github.com/awesomebytes/gentoo_prefix_ci |
191 |
> >>> > [2] [19]https://github.com/awesomebytes/gentoo_prefix_ci_32b |
192 |
> |
193 |
> >>> Yours, |
194 |
> >>> Benda |
195 |
> |
196 |
> >> -- |
197 |
> |
198 |
> >> Sammy Pfeiffer |
199 |
> >> PhD Candidate at The Magic Lab within UTS. |
200 |
> |
201 |
> > -- |
202 |
> |
203 |
> > Sammy Pfeiffer |
204 |
> > PhD Candidate at The Magic Lab within UTS. |
205 |
> |
206 |
> |
207 |
> |
208 |
> References: |
209 |
> 1. mailto:sammypfeiffer@×××××.com |
210 |
> 2. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033 |
211 |
> 3. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082 |
212 |
> 4. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247 |
213 |
> 5. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433 |
214 |
> 6. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501 |
215 |
> 7. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576 |
216 |
> 8. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608 |
217 |
> 9. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681 |
218 |
> 10. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726 |
219 |
> 11. https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355 |
220 |
> 12. mailto:sammypfeiffer@×××××.com |
221 |
> 13. mailto:heroxbd@g.o |
222 |
> 14. mailto:sammypfeiffer@×××××.com |
223 |
> 15. https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f |
224 |
> 16. https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1 |
225 |
> 17. https://gitweb.gentoo.org/repo/proj/prefix.git/ |
226 |
> 18. https://github.com/awesomebytes/gentoo_prefix_ci |
227 |
> 19. https://github.com/awesomebytes/gentoo_prefix_ci_32b |
228 |
> |
229 |
> read_char: errno==EILSEQ; invalid byte sequence for UTF-8: |
230 |
-- |
231 |
Fabian Groffen |
232 |
Gentoo on a different level |