Re: [gentoo-project] Re: A plan for Gentoo

[Daniel Robbins]

Hi All,

Just want to follow up for anyone following this thread.

I did participate in my first council meeting! Thanks to all council
members for facilitating that.

I'm going to continue to work on becoming a non-committing developer with
my mentor, Zac Medico.

I also had about 30 mins to chat in #gentoo-portage and outlined a plan for
moving Portage into the next generation. I am going to work on both an
architecture, as well as a roadmap, with the supervision of Zac Medico and
contribution of Zac Medico and others. This will be a fairly detailed
document that is going to provide a path so that we can evolve the existing
Portage code base organically and with as little disruption as possible to
a multi-core-leveraging and very capable modular package manager.

There are a lot of people who care a ton about Gentoo and Portage but there
is a bit of a deadlock about what next steps to take with its evolution.
The conflict I referred to earlier in #gentoo-portage was about whether to
scrap our package manager code base entirely or continue to evolve it.
There is a lot of passion around this issue. I didn't want to give too many
details about the conversations since the conversations I had were in
private, and the gossipy details aren't really relevant to the technical
effort. But I don't think I gave enough context so it sounded to some that
I was "causing trouble" by stirring up a hornet's nest. My apologies --
that was not my intention.

Back to the technical stuff. The proposal will attempt to strike a
reasonable middle ground -- modularization of the current code base, which
will start by creating two packages -- "ebuild" -- a PMS-compliant build
system, and "emerge" -- the upper-half dependency resolver and associated
tools, between which there will be a clearly documented API. This will both
allow us to play around with potential ebuild replacements (with no
commitment -- these can be done as side projects and we can decide to adopt
one once it proves its merit) as well as more capable multi-threaded emerge
replacements. While it may be nice to consider focusing development on
replacing ebuild, we need it continue to function and a rewrite will just
introduce too many annoying build bugs for users. So it's better to
maintain it as-is until something better is adopted. In the transition, our
package manager can support BOTH formats and they can interoperate (an
ebuild can depend on a "newbuild", and vice versa.) allowing for a gradual
and smooth transition of repos (main tree, overlays, projects, other
distros) to the new format. The separation of ebuild/emerge also opens up
the potential for use of some parts of non-official build systems by
hooking into our API, such as pkgcore and paludis, as desired. (While this
won't be part of our official plan, interested parties who want to pursue
this work will be supported.)

I think the one goal everyone can agree on is that sophisticated dependency
resolution does stand to benefit greatly from multi-core processors, so
there will be a roadmap to evolve the top half (emerge) so that it is
multi-core capable. This will essentially be a rewrite of emerge. In the
mean-time, the single-threaded version will continue to be maintained. The
multi-threaded version will be done in-house (in Gentoo-land.) I will also
try to specify some useful functionality for the top-half that I think we
should incorporate into our next-generation architecture. This may involve
the definition of a new spec so that pkgcore and paludis can incorporate
and leverage this functionality as well.

I will plan to incorporate feedback from developers and in particular
current and former Portage developers so that concerns are addressed. I'm
building the general skeleton of the architecture -- there will be quite a
bit that will be left to flesh out by others. I have been thinking about
how to tackle Portage's monolithic code base for years and I want to get my
ideas formalized and written down. It's exciting to think that we could be
finally doing this :)

I am sharing this all in the interest of transparency and will continue to
be transparent and communicate regularly about progress. Realistically I
will not be able to start this effort until the new year. I wish you all a
great new year and holiday season, and look forward some awesome
collaboration in 2018!

For the sake of keeping gentoo-project on topic, consider this a simple
status update of what drobbins is up to and how he's collaborating with
Gentoo, so that there is no mystery. Any replies related directly to the
Portage proposal -- whether you like the ideas, whether you want me to
contribute or not, whether you dislike pkgcore or paludis being mentioned
or not, whether you like the idea of splitting portage into ebuild/emerge
or not, whether repoman demonstrates that this is a bad idea or not,
whether you want to write it in rust, C++ or go or not -- that conversation
doesn't belong on this list. Nor is it worth engaging in technical
conversation until I have a document that is ready for public review. I
will share ideas and gather feedback informally in #gentoo-portage as I
work on the document. Consider all of this a fun thing to reflect on over
the holidays :)

Take care, everyone --

Daniel

On Sat, Dec 9, 2017 at 3:39 PM, Daniel Robbins <drobbins@××××××.org> wrote:

> Hey Alex,
>
> Yes, correct, it doesn't involve you :) I am aware of some dynamics that
> you are not.
>
> I do not want to alarm anyone -- I don't think things are about to
> implode. But I do bring up my areas of suggestion for a reason. They are
> based on facts and first-hand conversations. I was asked by Ulrich if I was
> aware of some dysfunction related to #gentoo-portage -- I wouldn't have
> brought it up if not asked directly -- but I definitely am aware. It does
> not manifest in the channel in any obvious way, but it's there and it
> involves valuable people to the project, on both sides, and some amount of
> animosity. Rather than focus on the problem, I would rather focus on the
> solution, and I think the solution is to positively engage with and have a
> plan for senior developers, one that creates enthusiasm and positive
> movement, eases some pent-up frustration points for various people and
> supports people in making a difference. I am trying to assist where I can.
> But we need to be able to acknowledge the problem before we can actually
> have any hope of addressing it.
>
> Again, the sky is not falling and I see an overwhelming amount of positive
> direction in regards to Gentoo as a whole. But I also do see areas of
> improvement. Gentoo has not 'arrived' at perfection. Yet the impossibility
> of perfection isn't an excuse to not tackle the current challenges on the
> horizon. At least, that is my opinion.
>
> My point is simply that yes, my observations are real and I think
> pertinent to the continued positive direction of the project. I'm not
> saying that to spread FUD -- but instead to suggest that Gentoo could be so
> much better if we found a way for the project to, say, engage with vapier
> in a positive way rather than have him leave in frustration, as one example
> (and frankly a big enough example that it should suffice to provide ample
> evidence for the reality of what I am saying to anyone who questions its
> existence.)
>
> I am looking at what we could be if we didn't just accept these incidents
> as inevitable and acceptable losses but looked at them as challenges to
> overcome and make us better for the next time. The idea of doing that is
> something I find personally exciting.
>
> -Daniel
>
> On Sat, Dec 9, 2017 at 1:25 PM, Alexander Berntsen <bernalex@g.o>
> wrote:
>
>> On 09/12/17 18:29, Daniel Robbins wrote:
>> > And yes, there is some conflict in regards to the future direction of
>> > Portage, but this conflict is not really out in the open.
>> There is? Speaking as the former lead dev, who just asked the current
>> lead dev (Brian) -- we in the Portage team don't know of any conflict
>> about "the future direction of Portage," so if there is a secret plot
>> going on somewhere, it doesn't involve us.
>>
>> --
>> Alexander
>> bernalex@g.o
>> https://secure.plaimi.net/~alexander
>>
>>
>