Re: [gentoo-soc] My GSoC Gentoo 2018 Proposal for "Portage powered Android"

[Pengcheng Xu]

Hi Benda,

Thanks for the feedback!

I’ve filled out the application and shared the link to Gentoo Foundation. The link is here
for your convenience:

https://docs.google.com/document/d/1v3yA4rkex5DGiPmdlXOSse5QZrZAL8QDJFNY2fFAMRY/edit?usp=sharing

> H30/03/18 9:34、Benda Xu <heroxbd@g.o>のメール:
>
> Hi Pengcheng,
>
> Very well written proposal indeed!  I enjoyed reading it.
>
> Pengcheng Xu <i@××××××××.moe> writes:
>
>> After getting in touch with the mentor of the idea, Benda Xu (heroxbd)
>> and exchanging ideas, I’ve made a draft on my ideas and a rough plan
>> about what I’m going to do, forming a proposal.  It’s available as PDF
>> in the attachment. Feedbacks wanted!
>
>> Gentoo GSoC 2018 Proposal ​
>> [draft]
>> Portage powered Android
>
> Just a random thought, you might want to revise the project name to make
> it more catchy.

I believe that “Portage powered Android” sounds like a good name for this. I’m open to
new, catchy names as well!

>
>> Idea overview:
>
>> ...from Google Summer of Code/2018/Ideas/Portage powered Android (by
>> heroxbd): The Android custom ROM development has been based on
>> cross-compilation and flashing whole partitions. This paradigm has
>> been serving well for the embedded system developments. But as the
>> performance of personal mobile devices boost, it becomes feasible and
>> desirable to introduce package management like personal
>> computers. Package management will make software installation and
>> update reliable, reproducible, incremental and convenient.
>
>> This project aims to introduce Gentoo's prestiges package manager,
>> portage, to manage Android software stack. We are going to use the
>> Gentoo on Android as a starting point.  Starting from the GNU userland
>> provided, we are going to work with the LineageOS build system based
>> on the Android Open Source Project, to write ebuilds for the
>> individual components. Starting from the linux kernel first, the
>> Android will be reproduced from bottom up.
>
> It will become better if you could reduce the quote into 1-2 sentences
> and rewrite the overview in your own words.

Rewritten in draft 2. Feedback wanted.

>
> Also, if you need to get familar with AOSP build system for this
> project, better get preapred now.

I’ve successfully ran the build process for Lineage OS and fixed some minor problems (most of them
due to misconfigured environment); have read through [this](https://elinux.org/Android_Build_System <https://elinux.org/Android_Build_System>)
and [this](https://wiki.lineageos.org/devices/angler/build <https://wiki.lineageos.org/devices/angler/build>) for instructions
as well, and I believe that’s enough to get started.

>
>> A more detailed look:
>
>> 1. The aim of this project, from my perspective of view, is to
>> completely rebuild Android from bottom up, starting with getting hand
>> on the raw, untouched build system of Android (or more specifically,
>> its Lineage OS distribution) and produce a working system, so that we
>> can verify that everything is working (unlikely), or fix up whatever
>> problems (very likely) we may run into when building up the Lineage
>> (large part of it AOSP) tree. Most importantly, this can get us
>> familiar with how the existing Android build system works,
>
> us == traditional GNU/Linux distribution community?  Because getting
> familiar with the AOSP is not an interest for the AOSP community.

Fixed. What I really meant here is that getting familiar with the build system is fundamental
for dissecting it afterwards.

>
>> while in the meantime getting familiar with the various tools to work
>> with Android, from cross-compile toolchains, deploying tools to
>> maintenance tools on the device (e.g. bootloader & recovery).
>
>> 2. Then, to gain control of the system and unleash the full power of
>> our GNU tools, I believe it necessary to place ourselves a layer ​below
>> ​the existing Android system, so that we can easily poke around in the
>> Android system in order to update it the way we want (the main purpose
>> of this project). I’ve come up with two approaches (call it Approach a
>> and b). Both of which require tampering the initramfs (maybe more),
>> but does not involve touching the whole bunch of mess inside Android:
>
>> a. From the entry point in the initramfs, we launch a small loader to
>> fork our real init into background,
>
> Is it possible just to have our real init call the Android init?  Then
> no background forking is needed.

That’s possible as well. In fact, IMO that’s nothing different from having a helper launching the two inits
and have our Linux init wait for the Android init; we do have to wait in both cases, after all.

>
>> and then fires up the Android init. Our launched init waits for
>> Android init finishing initializing the devices, then we can continue
>> our job, with the rest of the Android system booting up in the
>> meantime. (This idea originally came from this thread ​[1]​;
>
> s/this thread/this page/ ?

Fixed.

>
>> the difference is that we’ll be trying to launch a full-blown init
>> system, so that a great amount of packages in the existing Portage
>> tree can utilize their service files (whether systemd units or OpenRC
>> scripts) without modifying.
>
> s/modifying/modification/ .

Fixed.

>
>> b. Launch the system with our own (real) init, initializing the core,
>> necessary devices with udev (as what a normal Linux system will do),
>> then launch the Android (sub)system inside a container. This is
>> actually the technique used in the Android subsystem in Chrome OS ​[2]​,
>> with the help of various namespace facilities provided by the Linux
>> kernel.
>
> Also ARC++,
>
>  https://en.wikipedia.org/wiki/Google_App_Runtime_for_Chrome
>  https://lwn.net/Articles/701964/
>
> I am not sure if ARC++ needs to execute Android init.  Have you checked
> that?

I haven’t found much information regarding ARC++ other than the few write-ups and presentation
about the talk at XDC 2016. According to my predictions, however, I believe Android init is needed,
however, as the aim for ARC++ is to minimize Android Framework changes, so it should require the
conventional Android services as well (SurfaceFlinger, etc.); this is the case in the Graphics Overview
in the presentation for ARC++.

>
>> This approach looks more promising and with far less hacky-smells, yet
>> it definitely requires more investigation and work.
>
> If a central idea needs investigation, better do it now.

Fixed.

>
>> 3. After we have acquired the level of control and, meanwhile, have
>> the Android environment to deal with running somehow separated from
>> our existing Linux system (it’s a complete Linux system at this
>> point)
>
> GNU system?  GNU/Linux system?  This terminology can be comfusing
> because Android itself is also on top of Linux.

Fixed. Though IMO something built on top of Linux kernel can’t be called a Linux system thereby.

>
>> , we can start to move things inside Android territory into the
>> management of Portage.
>
>> I’ve splitted this process into the following stages:
>
>> a. Get the Android kernel source sorted out and packaged as some sort
>> of kernel source package. The final form should be something like
>> sys-kernel/android-sources with ​USE_EXPAND like ​PLATFORM​, which
>> denotes the variant of the kernel source according to different
>> devices (like ​PLATFORM=”hammerhead”​); or something like
>> android-kernel/hammerhead-sources​. This enables us to manage the
>> source and build process of the kernel more easily, as well as
>> tweaking options and applying patches. (maybe merging mainline kernel
>> as well, but that’s another story)
>
> Nice!
>
>> b. Try building some parts of the Android system (or more precisely,
>> some components) separately, and get a more detailed understanding of
>> the Android build system (namely ​Android.mk and friends). This process
>> is necessary, as the original Android build system is somehow a tied
>> piece of machinery, which is designed to work as a whole. Considerable
>> amount of work is needed to take this system apart, along with (maybe
>> quite a lot of) investigating, experimenting and hacking.
>
> If decoupling is difficult, building the whole AOSP from a single ebuild
> is also acceptable.  I can split them up later.

I’ll focus on decoupling, and if I ran into problems I would definitely need help as well.

>
>> c. After we have got a complete understanding of the Android build
>> system, as well as having been able to carve a part of it out without
>> much difficulties, we should start composing something like
>> ​android.eclass​, so that we can automate the process in handling the
>> Android components with Portage. The advantage of this stage is that
>> once this eclass is built, we can automate the process of building any
>> possible new or changed Android components, as well as reducing the
>> hacking work needed for each individual component of the Android
>> system. (Needing to hack every piece sounds like an impossible task.)
>
> Indeed.
>
>> d. The only job left for now would be to adopt every part of the
>> Android system to using the freshly-made eclass. This is also an
>> intensive testing process for the eclass, as surely many bugs and
>> uncaught corner cases will pop out here and there. This can enable us
>> to refine the eclass further. Another important thing at this part is
>> to make sure some sort of isolation in different versions of the core
>> system, as there’s a compatibility guarantee for Android applications
>> called API level. We need to make sure that we leverage Portage
>> techniques such as profiles to isolate different API levels apart, so
>> as not to mix the system up across different versions, thus breaking
>> the compatibility guarantee. Meanwhile, implementing this level of
>> isolation with profiles allows us to switch the profile and then roll
>> the system up to the new API level (no annoying blockers as that’s
>> somehow unrelated to the “base system” that’s used to compile the
>> Android land).
>
> Exactly.
>
>> 4. After 3, the main Android system would be sitting inside the
>> container / chroot safe and sound
>
> If the Android is in /system, is a container still necessary?

I consider it necessary, as we may run into issues caused by Android and GNU/Linux being inherently
different in managing the system (networking, device management, mounting, etc.). With namespaces,
we can reduce the probability of such things happening. Another consideration is that we don’t want the
environment of Android to be mixed up with GNU/Linux, as having one platform’s tools (shell,
coreutils, toolboxes, etc.) floating around in another platform’s environment can cause surprises for
applications (like something that expects GNU date being greeted by the trimmed version from Android
toolbox).

>
>> , with all parts of it being managed by Portage. This stage 4, as in
>> my plan, would be to clean up the work and sort out a way to
>> conveniently distribute the work.  Required things would be:
>
>> a. Blog (intensively?) about the work done, so as to enable
>> reproducing the entire process from scratch (in case this project is
>> later abandoned and forgotten in the dust, yet someone (?) from the
>> future wants to pick it up) -I’ve had quite unpleasant memories from
>> experience with Gentoo/FreeBSD, trying to guess how things worked from
>> void. Related documentation should be summed up and put into Gentoo
>> Wiki, perhaps.
>
> I prefer blogging happening along the work, instead of a series of final
> summaries.  That will give you early feedbacks.

Fixed.

>
>> b. Develop a way to distribute the initial setup process, as we can’t
>> expect every user to start with the bootstrap process. Alternatively,
>> offer a way to convert existing systems into the system we’re trying
>> to build in discussion. The first method is preferred, though, as we
>> reserve right to change the structure of the system instead of using
>> what’s determined by the existing Android system, such as partition
>> schemes, filesystem choices and filesystem root layouts.
>
> As the first step, please make sure a Gentoo-Handbook-like documentation
> is available.  Then we can talk about user-friendliness to the users
> outside Gentoo.

Fixed.

>
>> c. Set up build hosts for different devices, and (optionally) handle
>> the production of binary packages. Though this may sound contradicting
>> with the Gentoo philosophy, but building the Android system can be
>> tiring and resource-consuming, while requiring users to set up
>> cross-compile hosts sounds cumbersome. Or, at least, encourage the use
>> of a building hosts locally or at small scales.
>
> Same as the above.
>
>> Expected outcomes ​
>> (tl;dr)​
>> :
>
>> 1. What we get:
>
>> a. A working (perhaps no framebuffer support though; I believe that
>> belongs to another project as it would be distracting and too much
>> work)
>
> Avoid writing long sentence in paranthesis.  If you want to cover this
> point, write it out in the paragraph.

Fixed.

>
>
> IMHO, for me I am more interested in a working framebuffer than e.g., a
> binary package repository.  But you have the final word to prioritize
> the tasks.
>
> Alternatively to a traditional framebuffer, frecon might be insightful:
>
>  https://chromium.googlesource.com/chromiumos/platform/frecon/+/HEAD/DESIGN-DOC.md <https://chromium.googlesource.com/chromiumos/platform/frecon/+/HEAD/DESIGN-DOC.md>

Honestly speaking, I’m afraid that I don’t have the required skills to work with graphics stack. Looking back
 on Wayland on libhybris on Android, the intrinsics of Android EGL made it difficult to act as a Wayland server.
And, though ARC++ seems to be working out great, its scale seems to be enormous. I’m not quite optimistic
with this direction; rather than creating odds and ends that don’t work together well, I’d prefer something that’s
really achievable. Further discussions welcomed on this point.

>
>> Linux distribution on existing Android devices;
>
>> b. A working Android environment hosted inside or parallel with the
>> Linux environment, with Portage managing it, taking the place of the
>> original “System Upgrade” process;
>
> Nice!
>
>> c. A Portage overlay and ​android.eclass available to other general
>> purpose Gentoo Linux systems
>
> We can reuse proj/android.git from the beginning.
>
>  https://gitweb.gentoo.org/proj/android.git/ <https://gitweb.gentoo.org/proj/android.git/>

That’s nice to know about.

>
>> (though further adaption from projects like Android-x86, but that’s
>> out of the scope of this project)
>
> No, I don't think Android-x86 is relevant here.  It a dead-end and we
> don't need to mention about it.

Fine. Section revised.

>
>> 2. What we can do with this:
>
>> a. Utilize tools that we’re familiar with to do things on the mobile,
>> from more general toolchains available directly on the device, to
>> various services we may want to run; basically we have a portable
>> micro-server
>
> Absolutely!
>
> s/$/./
>
>> b. Smoothen the update process of system without having to load up a
>> full system image every time (this is especially important for Lineage
>> OS nightly users: you need a full rom package to flash even if only a
>> few bits of the system (usually just a few new commits) got changed)
>
> Indeed.  But try to get rid of the long nested paranthesis.

Fixed.

>
>> c. Easier adoption to techniques we’re already familiar with to
>> perform tasks
>
>> About me: I’m Pengcheng Xu, a student currently studying in the School
>> of Electronics Engineering and Computer Science at Peking University,
>> China. Having used Gentoo for over 5 years and still currently using
>> Gentoo as my server system (it’s a SPARC I got from Brendan Horan
>> <​brendan@×××××.hk​>​on the ​gentoo-dev​mailing list) as well as Gentoo
>> Prefix on macOS and Ubuntu Linux, I have proposed quite many bugs and
>> some patches as well
>
> Put a list of bugs and patches you worked with in an appendix
> explicitly.  If you have patches at hand, submit them soon.  If you have
> pending fixes for Gentoo/FreeBSD, publish them via the bugzilla.  This
> will make you more appealing to this program.

I’ve opened up an Appendix section for links to what I’ve done on the Bugzilla. Is that sufficient?

>
>
> Finally, please check out the student's guide to refine this proposal:
>
>  https://google.github.io/gsocguides/student/ <https://google.github.io/gsocguides/student/>

Got it. Thanks.

>
>> [1]: ​http://whiteboard.ping.se/Android/Debian
>> [2]:
>> https://events.static.linuxfound.org/sites/events/files/slides/
>> ContainersPresoLCE.pdf
>
> Yours,
> Benda


Pengcheng Xu
i@××××××××.moe