1 |
Hi Pengcheng,
|
2 |
|
3 |
Very well written proposal indeed! I enjoyed reading it.
|
4 |
|
5 |
Pengcheng Xu <i@××××××××.moe> writes:
|
6 |
|
7 |
> After getting in touch with the mentor of the idea, Benda Xu (heroxbd) |
8 |
> and exchanging ideas, I’ve made a draft on my ideas and a rough plan |
9 |
> about what I’m going to do, forming a proposal. It’s available as PDF |
10 |
> in the attachment. Feedbacks wanted! |
11 |
|
12 |
> Gentoo GSoC 2018 Proposal |
13 |
> [draft] |
14 |
> Portage powered Android |
15 |
|
16 |
Just a random thought, you might want to revise the project name to make
|
17 |
it more catchy.
|
18 |
|
19 |
> Idea overview: |
20 |
|
21 |
> ...from Google Summer of Code/2018/Ideas/Portage powered Android (by |
22 |
> heroxbd): The Android custom ROM development has been based on |
23 |
> cross-compilation and flashing whole partitions. This paradigm has |
24 |
> been serving well for the embedded system developments. But as the |
25 |
> performance of personal mobile devices boost, it becomes feasible and |
26 |
> desirable to introduce package management like personal |
27 |
> computers. Package management will make software installation and |
28 |
> update reliable, reproducible, incremental and convenient. |
29 |
|
30 |
> This project aims to introduce Gentoo's prestiges package manager, |
31 |
> portage, to manage Android software stack. We are going to use the |
32 |
> Gentoo on Android as a starting point. Starting from the GNU userland |
33 |
> provided, we are going to work with the LineageOS build system based |
34 |
> on the Android Open Source Project, to write ebuilds for the |
35 |
> individual components. Starting from the linux kernel first, the |
36 |
> Android will be reproduced from bottom up. |
37 |
|
38 |
It will become better if you could reduce the quote into 1-2 sentences
|
39 |
and rewrite the overview in your own words.
|
40 |
|
41 |
Also, if you need to get familar with AOSP build system for this
|
42 |
project, better get preapred now.
|
43 |
|
44 |
> A more detailed look: |
45 |
|
46 |
> 1. The aim of this project, from my perspective of view, is to |
47 |
> completely rebuild Android from bottom up, starting with getting hand |
48 |
> on the raw, untouched build system of Android (or more specifically, |
49 |
> its Lineage OS distribution) and produce a working system, so that we |
50 |
> can verify that everything is working (unlikely), or fix up whatever |
51 |
> problems (very likely) we may run into when building up the Lineage |
52 |
> (large part of it AOSP) tree. Most importantly, this can get us |
53 |
> familiar with how the existing Android build system works, |
54 |
|
55 |
us == traditional GNU/Linux distribution community? Because getting
|
56 |
familiar with the AOSP is not an interest for the AOSP community.
|
57 |
|
58 |
> while in the meantime getting familiar with the various tools to work |
59 |
> with Android, from cross-compile toolchains, deploying tools to |
60 |
> maintenance tools on the device (e.g. bootloader & recovery). |
61 |
|
62 |
> 2. Then, to gain control of the system and unleash the full power of |
63 |
> our GNU tools, I believe it necessary to place ourselves a layer below |
64 |
> the existing Android system, so that we can easily poke around in the |
65 |
> Android system in order to update it the way we want (the main purpose |
66 |
> of this project). I’ve come up with two approaches (call it Approach a |
67 |
> and b). Both of which require tampering the initramfs (maybe more), |
68 |
> but does not involve touching the whole bunch of mess inside Android: |
69 |
|
70 |
> a. From the entry point in the initramfs, we launch a small loader to |
71 |
> fork our real init into background, |
72 |
|
73 |
Is it possible just to have our real init call the Android init? Then
|
74 |
no background forking is needed.
|
75 |
|
76 |
> and then fires up the Android init. Our launched init waits for |
77 |
> Android init finishing initializing the devices, then we can continue |
78 |
> our job, with the rest of the Android system booting up in the |
79 |
> meantime. (This idea originally came from this thread [1]; |
80 |
|
81 |
s/this thread/this page/ ?
|
82 |
|
83 |
> the difference is that we’ll be trying to launch a full-blown init |
84 |
> system, so that a great amount of packages in the existing Portage |
85 |
> tree can utilize their service files (whether systemd units or OpenRC |
86 |
> scripts) without modifying. |
87 |
|
88 |
s/modifying/modification/ .
|
89 |
|
90 |
> b. Launch the system with our own (real) init, initializing the core, |
91 |
> necessary devices with udev (as what a normal Linux system will do), |
92 |
> then launch the Android (sub)system inside a container. This is |
93 |
> actually the technique used in the Android subsystem in Chrome OS [2], |
94 |
> with the help of various namespace facilities provided by the Linux |
95 |
> kernel. |
96 |
|
97 |
Also ARC++,
|
98 |
|
99 |
https://en.wikipedia.org/wiki/Google_App_Runtime_for_Chrome
|
100 |
https://lwn.net/Articles/701964/
|
101 |
|
102 |
I am not sure if ARC++ needs to execute Android init. Have you checked
|
103 |
that?
|
104 |
|
105 |
> This approach looks more promising and with far less hacky-smells, yet |
106 |
> it definitely requires more investigation and work. |
107 |
|
108 |
If a central idea needs investigation, better do it now.
|
109 |
|
110 |
> 3. After we have acquired the level of control and, meanwhile, have |
111 |
> the Android environment to deal with running somehow separated from |
112 |
> our existing Linux system (it’s a complete Linux system at this |
113 |
> point) |
114 |
|
115 |
GNU system? GNU/Linux system? This terminology can be comfusing
|
116 |
because Android itself is also on top of Linux.
|
117 |
|
118 |
> , we can start to move things inside Android territory into the |
119 |
> management of Portage. |
120 |
|
121 |
> I’ve splitted this process into the following stages: |
122 |
|
123 |
> a. Get the Android kernel source sorted out and packaged as some sort |
124 |
> of kernel source package. The final form should be something like |
125 |
> sys-kernel/android-sources with USE_EXPAND like PLATFORM, which |
126 |
> denotes the variant of the kernel source according to different |
127 |
> devices (like PLATFORM=”hammerhead”); or something like |
128 |
> android-kernel/hammerhead-sources. This enables us to manage the |
129 |
> source and build process of the kernel more easily, as well as |
130 |
> tweaking options and applying patches. (maybe merging mainline kernel |
131 |
> as well, but that’s another story) |
132 |
|
133 |
Nice!
|
134 |
|
135 |
> b. Try building some parts of the Android system (or more precisely, |
136 |
> some components) separately, and get a more detailed understanding of |
137 |
> the Android build system (namely Android.mk and friends). This process |
138 |
> is necessary, as the original Android build system is somehow a tied |
139 |
> piece of machinery, which is designed to work as a whole. Considerable |
140 |
> amount of work is needed to take this system apart, along with (maybe |
141 |
> quite a lot of) investigating, experimenting and hacking. |
142 |
|
143 |
If decoupling is difficult, building the whole AOSP from a single ebuild
|
144 |
is also acceptable. I can split them up later.
|
145 |
|
146 |
> c. After we have got a complete understanding of the Android build |
147 |
> system, as well as having been able to carve a part of it out without |
148 |
> much difficulties, we should start composing something like |
149 |
> android.eclass, so that we can automate the process in handling the |
150 |
> Android components with Portage. The advantage of this stage is that |
151 |
> once this eclass is built, we can automate the process of building any |
152 |
> possible new or changed Android components, as well as reducing the |
153 |
> hacking work needed for each individual component of the Android |
154 |
> system. (Needing to hack every piece sounds like an impossible task.) |
155 |
|
156 |
Indeed.
|
157 |
|
158 |
> d. The only job left for now would be to adopt every part of the |
159 |
> Android system to using the freshly-made eclass. This is also an |
160 |
> intensive testing process for the eclass, as surely many bugs and |
161 |
> uncaught corner cases will pop out here and there. This can enable us |
162 |
> to refine the eclass further. Another important thing at this part is |
163 |
> to make sure some sort of isolation in different versions of the core |
164 |
> system, as there’s a compatibility guarantee for Android applications |
165 |
> called API level. We need to make sure that we leverage Portage |
166 |
> techniques such as profiles to isolate different API levels apart, so |
167 |
> as not to mix the system up across different versions, thus breaking |
168 |
> the compatibility guarantee. Meanwhile, implementing this level of |
169 |
> isolation with profiles allows us to switch the profile and then roll |
170 |
> the system up to the new API level (no annoying blockers as that’s |
171 |
> somehow unrelated to the “base system” that’s used to compile the |
172 |
> Android land). |
173 |
|
174 |
Exactly.
|
175 |
|
176 |
> 4. After 3, the main Android system would be sitting inside the |
177 |
> container / chroot safe and sound |
178 |
|
179 |
If the Android is in /system, is a container still necessary?
|
180 |
|
181 |
> , with all parts of it being managed by Portage. This stage 4, as in |
182 |
> my plan, would be to clean up the work and sort out a way to |
183 |
> conveniently distribute the work. Required things would be: |
184 |
|
185 |
> a. Blog (intensively?) about the work done, so as to enable |
186 |
> reproducing the entire process from scratch (in case this project is |
187 |
> later abandoned and forgotten in the dust, yet someone (?) from the |
188 |
> future wants to pick it up) -I’ve had quite unpleasant memories from |
189 |
> experience with Gentoo/FreeBSD, trying to guess how things worked from |
190 |
> void. Related documentation should be summed up and put into Gentoo |
191 |
> Wiki, perhaps. |
192 |
|
193 |
I prefer blogging happening along the work, instead of a series of final
|
194 |
summaries. That will give you early feedbacks.
|
195 |
|
196 |
> b. Develop a way to distribute the initial setup process, as we can’t |
197 |
> expect every user to start with the bootstrap process. Alternatively, |
198 |
> offer a way to convert existing systems into the system we’re trying |
199 |
> to build in discussion. The first method is preferred, though, as we |
200 |
> reserve right to change the structure of the system instead of using |
201 |
> what’s determined by the existing Android system, such as partition |
202 |
> schemes, filesystem choices and filesystem root layouts. |
203 |
|
204 |
As the first step, please make sure a Gentoo-Handbook-like documentation
|
205 |
is available. Then we can talk about user-friendliness to the users
|
206 |
outside Gentoo.
|
207 |
|
208 |
> c. Set up build hosts for different devices, and (optionally) handle |
209 |
> the production of binary packages. Though this may sound contradicting |
210 |
> with the Gentoo philosophy, but building the Android system can be |
211 |
> tiring and resource-consuming, while requiring users to set up |
212 |
> cross-compile hosts sounds cumbersome. Or, at least, encourage the use |
213 |
> of a building hosts locally or at small scales. |
214 |
|
215 |
Same as the above.
|
216 |
|
217 |
> Expected outcomes |
218 |
> (tl;dr) |
219 |
> : |
220 |
|
221 |
> 1. What we get: |
222 |
|
223 |
> a. A working (perhaps no framebuffer support though; I believe that |
224 |
> belongs to another project as it would be distracting and too much |
225 |
> work) |
226 |
|
227 |
Avoid writing long sentence in paranthesis. If you want to cover this
|
228 |
point, write it out in the paragraph.
|
229 |
|
230 |
|
231 |
IMHO, for me I am more interested in a working framebuffer than e.g., a
|
232 |
binary package repository. But you have the final word to prioritize
|
233 |
the tasks.
|
234 |
|
235 |
Alternatively to a traditional framebuffer, frecon might be insightful:
|
236 |
|
237 |
https://chromium.googlesource.com/chromiumos/platform/frecon/+/HEAD/DESIGN-DOC.md
|
238 |
|
239 |
> Linux distribution on existing Android devices; |
240 |
|
241 |
> b. A working Android environment hosted inside or parallel with the |
242 |
> Linux environment, with Portage managing it, taking the place of the |
243 |
> original “System Upgrade” process; |
244 |
|
245 |
Nice!
|
246 |
|
247 |
> c. A Portage overlay and android.eclass available to other general |
248 |
> purpose Gentoo Linux systems |
249 |
|
250 |
We can reuse proj/android.git from the beginning.
|
251 |
|
252 |
https://gitweb.gentoo.org/proj/android.git/
|
253 |
|
254 |
> (though further adaption from projects like Android-x86, but that’s |
255 |
> out of the scope of this project) |
256 |
|
257 |
No, I don't think Android-x86 is relevant here. It a dead-end and we
|
258 |
don't need to mention about it.
|
259 |
|
260 |
> 2. What we can do with this: |
261 |
|
262 |
> a. Utilize tools that we’re familiar with to do things on the mobile, |
263 |
> from more general toolchains available directly on the device, to |
264 |
> various services we may want to run; basically we have a portable |
265 |
> micro-server |
266 |
|
267 |
Absolutely!
|
268 |
|
269 |
s/$/./
|
270 |
|
271 |
> b. Smoothen the update process of system without having to load up a |
272 |
> full system image every time (this is especially important for Lineage |
273 |
> OS nightly users: you need a full rom package to flash even if only a |
274 |
> few bits of the system (usually just a few new commits) got changed) |
275 |
|
276 |
Indeed. But try to get rid of the long nested paranthesis.
|
277 |
|
278 |
> c. Easier adoption to techniques we’re already familiar with to |
279 |
> perform tasks |
280 |
|
281 |
> About me: I’m Pengcheng Xu, a student currently studying in the School |
282 |
> of Electronics Engineering and Computer Science at Peking University, |
283 |
> China. Having used Gentoo for over 5 years and still currently using |
284 |
> Gentoo as my server system (it’s a SPARC I got from Brendan Horan |
285 |
> <brendan@×××××.hk>on the gentoo-devmailing list) as well as Gentoo |
286 |
> Prefix on macOS and Ubuntu Linux, I have proposed quite many bugs and |
287 |
> some patches as well |
288 |
|
289 |
Put a list of bugs and patches you worked with in an appendix
|
290 |
explicitly. If you have patches at hand, submit them soon. If you have
|
291 |
pending fixes for Gentoo/FreeBSD, publish them via the bugzilla. This
|
292 |
will make you more appealing to this program.
|
293 |
|
294 |
|
295 |
Finally, please check out the student's guide to refine this proposal:
|
296 |
|
297 |
https://google.github.io/gsocguides/student/
|
298 |
|
299 |
> [1]: http://whiteboard.ping.se/Android/Debian |
300 |
> [2]: |
301 |
> https://events.static.linuxfound.org/sites/events/files/slides/ |
302 |
> ContainersPresoLCE.pdf |
303 |
|
304 |
Yours,
|
305 |
Benda |