[gentoo-portage-dev] New proposed modular sync system

[Brian Dolbec]

First off, I know many of you think portage needs to do everything on
it's own.  NO outside dependencies.

BUT!  Please hear me out.

While working on many of gentoo's tools gentoolkit's equery, eclean,
enalyze, Layman, mirrorselect, etc..  I have found that there was a lot
of needless code duplication.  Both within the apps themselves and
copying/re-inventing the nearly identical code in other apps.

I want to move many of gentoo's core applications to using some more
shared libraries.  But that in itself is not a topic to discuss in this
thread.  Please, do not sidetrack this thread.  The above is only for
background info leading to this proposed change.  This proposal follows
along that thinking.

Many years ago after re-writing ecleans code I began re-writing
portage's emaint code so that it could be imported and used internally
in eclean to re-generate the Packages binpkg index file.  In that code I
found a todo which said "make a plug-in system so we don't have to keep
modifying this code"  or something along those lines.  So I did.  It
took 4 years+, rebasing many times and migrating vcs's, but Zac finally
merged it.  It has been used in portage for more than a year. That
plug-in system I designed to be very flexible and basic.  It can be used
for many areas of portage to simplify it's code and the maintenance
required to keep current with changes in repository types, and transfer
methods.

I have started a Proposals sub-page under the Portage project page in
the wiki.  It has a link to a diagram I made showing how the plug-in
system is laid out.  This thread will be used to discuss the proposal
and the details needed for the module_spec dictionary that is used to
provide the manager with the details of what a module provides, options,
etc..

For a working example of the system in action, just try out emaint -h,
and some other actions it provides.  Then move one of it's modules out
of the the emaint/modules/ directory and re-try emaint -h.  You will see
the options for that module and it's actions are no longer available.
No code editing was required, no errors occurred,...  Put it back,
retry...

The modular sync system I propose would re-use that plug-in manager to
interface the sync manager code with the modules that perform the sync
actions.  This sytem is easily extended with new modules for different
transfer methods or VCS specific modules.

This also makes it possible for other gentoo applications like layman to
install a layman specific module which syncs all layman overlays that it
installs.  All this done with out effort on the part of the portage team
either for maintenance or code changes to support it's addition.  The
same can be said for Funtoo, if our git module is not suiotable, they
just install their own custom sync module.  Done :)

These add-on sync modules can also be installed via their own ebuild.

For example:  Zac has for years had a custom script that syncs the
portage tree and makes a squashfs version of it which he uses for his
PORTDIR. it is faster than a normal one.  He, or someone else could make
a app-portage/squashfs-sync ebuild that installs a module for that
purpose.  The user then just needs to edit his repos.conf file and
change the sync-type to squashfs.  Then emerge --sync will do it
automatically without user intervention and auxiliary scripts run
manually or cron job.

This system would also make it possible to modify emerge-s cli to accept
target repos to sync. and not any others also installed.  Similarly to
"layman -s some-overlay", "emerge --sync some-overlay"
 
P.S. sorry for such a long email 
-- 
Brian Dolbec <dolsen@g.o>