[gentoo-desktop] Re: Interest inquery: kde4-nosemantic overlay

[Duncan]

Martin Vaeth posted on Thu, 11 Jul 2013 07:25:37 +0000 as excerpted:

> Independently on the overlay, I think your framework is very
> interesting:
> Does your framework manage the ebuilds in some overlay, or is it
> actually running tranparently during emerge (probably with a patched
> version of portage), allowing e.g. to change metadata without
> maintaining the ebuild in a separate local overlay?
> (I guess that it is the former, but your choice of the path
> /etc/portage/... suggests the latter)

To be explicit, I'm using the official gentoo/kde overlay, which of 
course is the only place with kde 4.11 series (4.10.80+) ebuilds ATM, 
since 4.11 is still pre-release and we're dealing with the beta (4.10.80, 
4.10.90) and rc (4.10.95+) sources and ebuilds.  But the framework I've 
setup is repo agnostic and would find the ebuilds in whatever repo, main 
tree or active overlay, they appear in.

Some background...

For some years now some ebuilds have made use of the epatch_user function 
from eutils.eclass or base.eclass.  That function made use of a patches 
tree organized by category and package at /etc/portage/patches/, 
automatically applying any patches it found in the directory 
corresponding to the ebuild being run.  Originally, calling epatch_user 
was optional -- the ebuild had to inherit the appropriate eclass and call 
the function.  However, with eapi-5, it's now a mandatory part of any 
eapi-5 compliant ebuild.

Actually, I believe that idea originated with a guy (I've forgotten his 
name and AFAIK he's no longer a gentooer, IIRC his domain expired some 
time ago and I deleted the bookmark I had to it) who had a patched 
portage with hooks at selected spots in the code, and various optional 
utilities using those hooks for all sorts of useful stuff.  It seems 
enough gentoo devs found his stuff useful, that portage gradually 
integrated many of his hooks and tools, including this one.

Meanwhile, long before eapi-5 came along, I got tired of having to figure 
out whether dropping a patch in the /etc/portage/patches tree was going 
to work for a particular ebuild or not, and hacked up something using 
/etc/portage/bashrc and a couple stub scripts, that ensured that I had 
epatch_user available as a function, and ran it using one of the hooks 
portage integrated for such extensions, the post_src_prepare function, as 
I defined it in /etc/portage/bashrc.  So for some time now, I've been 
able to simply drop source patches in the appropriate /etc/portage/
patches/ subdir and have them automatically applied, regardless of 
whether the ebuild actually called epatch_user or not.

As anyone who for example tests gcc versions before they're unmasked will 
know, a patch tree of this nature comes in really handy for applying 
various source patches without having to manually modify the ebuild. =:^)

But, while that works great for source patches, at times one has to 
modify ebuilds too, and there isn't yet an offical similar framework for 
automatically applying ebuild patches.

I've occasionally been frustrated by this, but until this gentoo/kde 
semantic-desktop policy change, particularly with epatch_user eliminating 
the need for most of the ebuild edits I used to do, it was just easier to 
do the manual hacking any time I needed to change an actual ebuild.

But with the gentoo/kde semantic-desktop policy change, that too changed, 
as I was now doing ebuild patching longer term and on a rather wider 
scale.  So I needed something like epatch_user, but for the ebuilds 
themselves instead of for the sources the ebuilds used.


Meanwhile, some time ago I setup a script that among other things ran 
emerge sync and layman -S (in parallel), then ran emerge --regen (with 
multiple jobs) to rebuild the cache to take into account layman's 
overlays.  Over time, this script expanded a bit to take on additional 
tasks, including checking to see if my portage and sources partition was 
mounted before running the sync and mounting it if not, updating the 
esearch and portage-utils databases, doing an emerge --update --deep
--fetch @world, etc.  The script is called esyn (esync without the 
terminating c), and I run it instead of esync to automatically take care 
of all the stuff I'd otherwise do one command at a time at the beginning 
of an update run.


So when the need for an ebuild-patching variant of the epatch_user 
function came up with this gentoo/kde policy change and my subsequent 
generation of all these ebuild patches I needed applied, it was rather 
natural to simply add another function to my esyn script, that after
the syncs looped thru a tree paralleling /etc/portage/patches (which I 
chose to call /etc/portage/patches.ebuild), and upon finding a patch in 
that tree, looped thru each of the repos listed in $PORTDIR and 
$PORTDIR_OVERLAY (as set in make.conf), matching category and package to 
see if there are any matching ebuilds in that repo to patch.

If it finds a matching ebuild, like epatch_user, it first tries applying 
the patch in a dry-run.  If the patch applies in the dry-run, then it 
applies it for real.  If the patch doesn't apply in the dry-run, then it 
could be an ebuild for an old version (in this case, kde 4.10 ebuilds 
still have the semantic-desktop USE flag and don't need patched, neither 
will the patches apply), or it might be an ebuild that had the patch 
applied already (unlike with rsync, unless there's a conflict, git 
doesn't overwrite local changes, so the patch stays applied at layman -S 
and an attempted re-apply will fail; if there's a conflict, I can git 
reset --hard the overlay and redo the overlay sync to pull down the 
update, after which, given the conflict with the patch I had previously 
applied, I'll likely need to create an updated patch to apply), etc, so 
the function simply skips that ebuild and moves on to the next.

At this point it's all rather hacked together, but it works here.  I 
expect that over time, as the ebuilds from the gentoo/kde overlay and 
eventually in the main tree change, I'll find the patches don't apply and 
my updates break, at which point I'll update the patches and/or update 
the ebuild patching function as necessary.  Based on past experience, on 
my own I'd get something semi-robust for my own setup in a few months to 
a year, tho it still wouldn't necessarily work well for others.  Of 
course if I were to turn the whole thing into a publicly available 
project and take patches or even make it a publicly managed project (much 
like kde-sunset), I expect it'd mature much faster and would be become 
rather less hacky and rather more general purpose relatively quickly, 
such that with the help of others it'd likely be generally usable and 
reasonably stable within a couple months, instead of the year or so it'll 
likely take me on my own, after which it'll be rather more robust but 
still be targeted at only my system, if I don't take it public.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman