1 |
Hey Fabian, |
2 |
|
3 |
Thanks for chipping in. I do think it's useful and user-friendly to be able |
4 |
to control the target of the interactive/noninteractive script. |
5 |
I proposed in my first email in the thread something similar with the |
6 |
$STOP_AFTER_STAGE |
7 |
variable, even though I was capturing it from the commandline, and as I |
8 |
understand |
9 |
you propose, it would probably be better to capture it from an environment |
10 |
variable (it would actually be more in-line with other existing variables). |
11 |
|
12 |
Benda, I'm sorry for the noise I'm causing. I just want to improve the |
13 |
experience for the next person that tries to use this amazing project by |
14 |
taking advantage of already implemented stuff. And also just having this |
15 |
discussion is good to have more information archived to google about Gentoo |
16 |
Prefix and its bootstrap process. I think. |
17 |
|
18 |
On Sun, 2 Feb 2020 at 23:08, Fabian Groffen <grobian@g.o> wrote: |
19 |
|
20 |
> What I gather from this discussion is that the bootstrap script is |
21 |
> complex, and grew more complexity over the years. |
22 |
> |
23 |
> Back in the early days, there was just documentation, one needed to |
24 |
> follow. Since this frequently needed an update, and people kept using |
25 |
> older copies, at some point the bootstrap-prefix.sh script was |
26 |
> introduced. |
27 |
> |
28 |
> This script followed the documentation to the letter. It was an exact |
29 |
> replica of the instructions in the bootstrap document. This included |
30 |
> the notions of 3 stages and a world recompile. |
31 |
> |
32 |
> Today, the purpose of the stages have blurred, and seem more like |
33 |
> batches in which to do things. Some of the initial intentions still |
34 |
> stand out, e.g. that stage1 is there to provide a working portage |
35 |
> environment with some minimal efforts. Minimal got redefined as soon as |
36 |
> we had to install python and more and more tools seemed necesary (sed, |
37 |
> path, awk, etc.). |
38 |
> |
39 |
> Now, it seems you need to cut time and sort of bootstrap up-to a stage. |
40 |
> Would it be an option to simply add a(nother) var that will control the |
41 |
> interactive target in terms of how far it will run along. Normal resume |
42 |
> strategies it employs already would simply allow a next run to run along |
43 |
> further. |
44 |
> |
45 |
> I think above can be done easily with a few lines of code. |
46 |
> |
47 |
> Let me know what you think. |
48 |
> |
49 |
> Thanks, |
50 |
> Fabian |
51 |
> |
52 |
> On 23-01-2020 13:31:05 +1100, Sam Pfeiffer wrote: |
53 |
> > Grobian, any input about this? |
54 |
> > |
55 |
> > On Thu, Jan 16, 2020, 17:39 Sam Pfeiffer <[1]sammypfeiffer@×××××.com> |
56 |
> wrote: |
57 |
> > |
58 |
> > > Hello Benda, |
59 |
> > |
60 |
> > > To be more specific... To easily go thru it I made a gist with the |
61 |
> latest |
62 |
> > > version of bootstrap-prefix.sh and I'll be pointing to specific lines |
63 |
> (this |
64 |
> > > also helps me recap why I did this, as I remember using stage{1,2,3} |
65 |
> didn't |
66 |
> > > work for me when I first came into this): |
67 |
> > |
68 |
> > > When you execute bootstrap-prefix.sh without arguments (other than the |
69 |
> > > EPREFIX, ./bootstrap-prefix.sh /MY/EPREFIX) it calls this line: |
70 |
> > |
71 |
> > > [2] |
72 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033 |
73 |
> > |
74 |
> > > Calling bootstrap_interactive |
75 |
> > |
76 |
> > > If you give an extra argument (./bootstrap-prefix.sh /MY/EPREFIX |
77 |
> > > [noninteractive, stage{1,2,3}]) it calls this line: |
78 |
> > > [3] |
79 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082 |
80 |
> > |
81 |
> > > bootstrap_${TODO#non} || exit 1 |
82 |
> > |
83 |
> > > Which, on the noninteractive case, goes to the same place than no |
84 |
> arguments |
85 |
> > > (bootstrap_interactive). But in the stage{1,2,3} calls directly the |
86 |
> > > bootstrap_stageX function. |
87 |
> > |
88 |
> > > The thing is, noninteractive does some checks, finds resources and |
89 |
> sets up |
90 |
> > > environment variables like: |
91 |
> > |
92 |
> > > * Check for bad flags: |
93 |
> > > [4] |
94 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247 |
95 |
> > > * We find a gcc: |
96 |
> > > [5] |
97 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433 |
98 |
> > > * We get a cpu count and adjust -j for Make: |
99 |
> > > [6] |
100 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501 |
101 |
> > > * Check if we deal with multilib systems: |
102 |
> > > [7] |
103 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576 |
104 |
> > > * Check if we are boostrapping from a Gentoo system: |
105 |
> > > [8] |
106 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608 |
107 |
> > > * Set our EPREFIX: |
108 |
> > > [9] |
109 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681 |
110 |
> > |
111 |
> > > Which is summarised at the end by setting the environment variables |
112 |
> EPREFIX, |
113 |
> > > CHOST, PATH, MAKEOPTS. |
114 |
> > |
115 |
> > > And then finally calls bootstrap_stage1_log, which runs |
116 |
> bootstrap_stage1 |
117 |
> > > itself. |
118 |
> > |
119 |
> > > [10] |
120 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726 |
121 |
> > |
122 |
> > > Then, we compare with calling bootstrap-prefix.sh EPREFIX stage1... |
123 |
> > |
124 |
> > > Calls bootstrap_stage1(): |
125 |
> > > [11] |
126 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355 |
127 |
> > |
128 |
> > > And as you can see it won't do any of the extra work done in |
129 |
> > > bootstrap_interactive(). |
130 |
> > |
131 |
> > > I'm not that versed in this script, so I may be missing something easy |
132 |
> to |
133 |
> > > overcome this (or I don't know enough about what is going on), but |
134 |
> there is no |
135 |
> > > separate function that does everything in bootstrap_interactive without |
136 |
> > > calling bootstrap_stage1. And bootstrap_stage1 does not do all that |
137 |
> setup that |
138 |
> > > bootstrap_interactive does. So that's how I came to my proposal. |
139 |
> > |
140 |
> > > If there is some patch to enable that setup from bootstrap_interactive |
141 |
> with |
142 |
> > > bootstrap_stage1, that would be a good alternative, I guess. I still |
143 |
> like the |
144 |
> > > fact that with my approach you are just saying "bootstrap everything |
145 |
> until |
146 |
> > > this step" instead of needing to manually do bootstrap_setup (imaginary |
147 |
> > > function I just named) then bootstrap_stage1, then bootstrap_stage2, |
148 |
> then |
149 |
> > > bootstrap_stage3, and then no argument for the last emerge -e system, |
150 |
> to get |
151 |
> > > to a specific bootstrap stage, but that's just commodity. |
152 |
> > |
153 |
> > > Let me know if I'm wrong somewhere or you think of any better solution! |
154 |
> > |
155 |
> > > On Thu, 16 Jan 2020 at 13:42, Sam Pfeiffer <[12] |
156 |
> sammypfeiffer@×××××.com> |
157 |
> > > wrote: |
158 |
> > |
159 |
> > >> Hi Benda, |
160 |
> > |
161 |
> > >> Thanks for your reply. |
162 |
> > |
163 |
> > >> On Thu, 16 Jan 2020 at 13:01, Benda Xu <[13]heroxbd@g.o> |
164 |
> wrote: |
165 |
> > |
166 |
> > >>> Hi Sam, |
167 |
> > |
168 |
> > >>> Sam Pfeiffer <[14]sammypfeiffer@×××××.com> writes: |
169 |
> > |
170 |
> > >>> > To ease automated building and debugging of the bootstrap of Gentoo |
171 |
> > >>> > Prefix I would like to propose a patch to add the option to stop |
172 |
> the |
173 |
> > >>> > non-interactive bootstrap on specific stages of the bootstrap. |
174 |
> > >>> > |
175 |
> > >>> > I'm currently doing this in my CI projects [1][2] and I'd love to |
176 |
> have |
177 |
> > >>> > this added upstream. |
178 |
> > >>> > |
179 |
> > >>> > I propose a patch like this: |
180 |
> > >>> >[15] |
181 |
> https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f |
182 |
> > >>> > |
183 |
> > >>> > Which is what I'm currently using. It just adds another optional |
184 |
> > >>> >parameter to the commandline (a 3rd one) in non-interactive mode |
185 |
> which |
186 |
> > >>> >you tell it after which stage to stop (stage1, stage2, stage3). |
187 |
> > |
188 |
> > >>> > I'm not the most versed person in bash, so maybe there is a more |
189 |
> > >>> > conceptually beautiful way of doing this. My patch is just a |
190 |
> proposal |
191 |
> > >>> > (that has been tested and running). |
192 |
> > >>> > |
193 |
> > >>> > This patch is useful because it allows: |
194 |
> > >>> > 1) Easier debugging by stages. |
195 |
> > |
196 |
> > >>> > 2) Allows to split the bootstrap job in parts in CI environments |
197 |
> with |
198 |
> > >>> > time limits on jobs to run (e.g. Azure Pipelines time limit is 6h*, |
199 |
> > >>> > and the bootstrap takes in between 5h40-6h40 depending on random |
200 |
> > >>> > things like time to download things, and load on the machine... or |
201 |
> > >>> > packages just taking longer to build). |
202 |
> > >>> > |
203 |
> > >>> > And this patch should not change anything to anyone else (AFAIK). |
204 |
> > |
205 |
> > >>> Are you aware that in bootstrap-prefix.sh you can specify the stage |
206 |
> you |
207 |
> > >>> are after? e.g. |
208 |
> > |
209 |
> > >>> [16] |
210 |
> https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1 |
211 |
> > |
212 |
> > >>> The text above that could be outdated, but PATH exports and stage1~3 |
213 |
> are |
214 |
> > >>> correct. |
215 |
> > |
216 |
> > >> I'm aware, but the behaviour (relating to environment variables and |
217 |
> choosing |
218 |
> > >> CPU numbers) differs from the |
219 |
> > |
220 |
> > >> noninteractive argument. I'd like to keep the CI environment as close |
221 |
> as a |
222 |
> > >> user executing ./bootstrap-prefix.sh EPREFIX as possible. |
223 |
> > |
224 |
> > >> That's why I'm proposing this addition. Another approach could be |
225 |
> modifying |
226 |
> > >> the current argument to do the same than the noninteractive one. |
227 |
> > |
228 |
> > >> But that may break the workflow of someone, and I don't want that. |
229 |
> > |
230 |
> > >> |
231 |
> > |
232 |
> > >>> > Thanks for your time and feedback! |
233 |
> > >>> > |
234 |
> > >>> > P.S.: I'm used to the GitHub/GitLab workflow of having a graphical |
235 |
> > >>> > interface to do a Pull Request... I don't know how to do this with |
236 |
> > >>> > [17]https://gitweb.gentoo.org/repo/proj/prefix.git/ If there is |
237 |
> no way, |
238 |
> > >>> > aside of the goal of this email, I'd propose to have a mirror in |
239 |
> > >>> > GitHub. |
240 |
> > |
241 |
> > >>> Yes, that's a good to have for me, but not in high priority. |
242 |
> > |
243 |
> > >>> > *If you are curious, Stage 1 takes 8min~, Stage 2 takes 29min~, |
244 |
> Stage |
245 |
> > >>> > 3 takes 2h30min~, emerge system takes 2h35min~ on Azure Pipelines, |
246 |
> > >>> > approximately. |
247 |
> > |
248 |
> > >>> Thank you so much for all your work! That's extremely helpful to |
249 |
> keep |
250 |
> > >>> the quality of Prefix codebase. |
251 |
> > |
252 |
> > >>> > [1] [18]https://github.com/awesomebytes/gentoo_prefix_ci |
253 |
> > >>> > [2] [19]https://github.com/awesomebytes/gentoo_prefix_ci_32b |
254 |
> > |
255 |
> > >>> Yours, |
256 |
> > >>> Benda |
257 |
> > |
258 |
> > >> -- |
259 |
> > |
260 |
> > >> Sammy Pfeiffer |
261 |
> > >> PhD Candidate at The Magic Lab within UTS. |
262 |
> > |
263 |
> > > -- |
264 |
> > |
265 |
> > > Sammy Pfeiffer |
266 |
> > > PhD Candidate at The Magic Lab within UTS. |
267 |
> > |
268 |
> > |
269 |
> > |
270 |
> > References: |
271 |
> > 1. mailto:sammypfeiffer@×××××.com |
272 |
> > 2. |
273 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3033 |
274 |
> > 3. |
275 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L3082 |
276 |
> > 4. |
277 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2200-L2247 |
278 |
> > 5. |
279 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2273-L2433 |
280 |
> > 6. |
281 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2443-L2501 |
282 |
> > 7. |
283 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2503-L2576 |
284 |
> > 8. |
285 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2578-L2608 |
286 |
> > 9. |
287 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2610-L2681 |
288 |
> > 10. |
289 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L2724-L2726 |
290 |
> > 11. |
291 |
> https://gist.github.com/awesomebytes/ba4df9c6ef3ab5742cb9aceed7998c3d#file-bootstrap-prefix-sh-L1355 |
292 |
> > 12. mailto:sammypfeiffer@×××××.com |
293 |
> > 13. mailto:heroxbd@g.o |
294 |
> > 14. mailto:sammypfeiffer@×××××.com |
295 |
> > 15. |
296 |
> https://gist.github.com/awesomebytes/3468477c6c90fe3d985372d50aabba9f |
297 |
> > 16. |
298 |
> https://wiki.gentoo.org/wiki/Project:Prefix/Manual_Bootstrap#Stage_1 |
299 |
> > 17. https://gitweb.gentoo.org/repo/proj/prefix.git/ |
300 |
> > 18. https://github.com/awesomebytes/gentoo_prefix_ci |
301 |
> > 19. https://github.com/awesomebytes/gentoo_prefix_ci_32b |
302 |
> > |
303 |
> > read_char: errno==EILSEQ; invalid byte sequence for UTF-8: |
304 |
> -- |
305 |
> Fabian Groffen |
306 |
> Gentoo on a different level |
307 |
> |
308 |
|
309 |
|
310 |
-- |
311 |
|
312 |
*Sammy Pfeiffer* |
313 |
PhD Candidate at The Magic Lab within UTS. |