Re: [gentoo-portage-dev] New proposed modular sync system

[Alec Warner]

On Thu, Jan 9, 2014 at 9:33 AM, Brian Dolbec <dolsen@g.o> wrote:

> First off, I know many of you think portage needs to do everything on
> it's own.  NO outside dependencies.
>
> BUT!  Please hear me out.
>
> While working on many of gentoo's tools gentoolkit's equery, eclean,
> enalyze, Layman, mirrorselect, etc..  I have found that there was a lot
> of needless code duplication.  Both within the apps themselves and
> copying/re-inventing the nearly identical code in other apps.
>
> I want to move many of gentoo's core applications to using some more
> shared libraries.  But that in itself is not a topic to discuss in this
> thread.  Please, do not sidetrack this thread.  The above is only for
> background info leading to this proposed change.  This proposal follows
> along that thinking.
>
> Many years ago after re-writing ecleans code I began re-writing
> portage's emaint code so that it could be imported and used internally
> in eclean to re-generate the Packages binpkg index file.  In that code I
> found a todo which said "make a plug-in system so we don't have to keep
> modifying this code"  or something along those lines.  So I did.  It
> took 4 years+, rebasing many times and migrating vcs's, but Zac finally
> merged it.  It has been used in portage for more than a year. That
> plug-in system I designed to be very flexible and basic.  It can be used
> for many areas of portage to simplify it's code and the maintenance
> required to keep current with changes in repository types, and transfer
> methods.
>
> I have started a Proposals sub-page under the Portage project page in
> the wiki.  It has a link to a diagram I made showing how the plug-in
> system is laid out.  This thread will be used to discuss the proposal
> and the details needed for the module_spec dictionary that is used to
> provide the manager with the details of what a module provides, options,
> etc..
>
> For a working example of the system in action, just try out emaint -h,
> and some other actions it provides.  Then move one of it's modules out
> of the the emaint/modules/ directory and re-try emaint -h.  You will see
> the options for that module and it's actions are no longer available.
> No code editing was required, no errors occurred,...  Put it back,
> retry...
>
> The modular sync system I propose would re-use that plug-in manager to
> interface the sync manager code with the modules that perform the sync
> actions.  This sytem is easily extended with new modules for different
> transfer methods or VCS specific modules.
>
> This also makes it possible for other gentoo applications like layman to
> install a layman specific module which syncs all layman overlays that it
> installs.  All this done with out effort on the part of the portage team
> either for maintenance or code changes to support it's addition.  The
> same can be said for Funtoo, if our git module is not suiotable, they
> just install their own custom sync module.  Done :)
>
> These add-on sync modules can also be installed via their own ebuild.
>
> For example:  Zac has for years had a custom script that syncs the
> portage tree and makes a squashfs version of it which he uses for his
> PORTDIR. it is faster than a normal one.  He, or someone else could make
> a app-portage/squashfs-sync ebuild that installs a module for that
> purpose.  The user then just needs to edit his repos.conf file and
> change the sync-type to squashfs.  Then emerge --sync will do it
> automatically without user intervention and auxiliary scripts run
> manually or cron job.
>
> This system would also make it possible to modify emerge-s cli to accept
> target repos to sync. and not any others also installed.  Similarly to
> "layman -s some-overlay", "emerge --sync some-overlay"
>
>
I think the opposition to this idea primarily falls on reliability. There
have been a number of hacks to portage over the years to keep it
functioning during upgrades of itself, even down to the non-standard place
where its .py files are stored (/usr/lib/portage/pym?) So the real question
is when we move to a plug-in architecture, how do we add, remove, upgrade
the plugins without breaking the actual package manager?

-A




> P.S. sorry for such a long email
> --
> Brian Dolbec <dolsen@g.o>
>