[gentoo-dev] Re: Sets in the tree

[Duncan]

Sergey Popov posted on Wed, 14 Aug 2013 17:07:32 +0400 as excerpted:

> Why [were sets] not added as a part of the PMS? Some implementation
> flaws? Or maybe, architecture problems?

[TL;DR folks, skip to last paragraph summary.]

(As a user who has been using sets as they appear in the kde overlay and 
portage 2.2 for quite some time... This is obviously my opinion based on 
what I've seen from my vantage-point, and is thus open to correction from 
those closer to the action.)

AFAIK, it's several things.

I believe the first sets implementation was in paludis, with the kde-
overlay an early user back when it was first setup and it required 
paludis.

When portage implemented sets (and the kde-overlay stopped requiring 
paludis and began using portage sets as an OPTION to the meta-packages 
that were also setup there and in the main tree), its implementation was 
somewhat different, which ended up being a bit awkward, because paludis 
had implemented them first and so had first-mover default, but portage 
remains gentoo default PM, but the implementations weren't (fully?) 
compatible, so it was a bit of a battle of the defaults.

I believe that's actually one of the big things that kept portage's sets 
implementation in experimental-masked 2.2 for so long -- at least at 
first, there was an intention to try to settle on a sets standard that 
both (plus pkgcore) could implement.  But over time it became more and 
more apparent that simply wasn't going to happen, and for quite some time 
I at least, thought the feature was doomed to be forever masked.

Meanwhile, the role of PMS itself became clearer and somewhat more 
strictly limited, over time.  Ciaranm or someone else can probably give a 
more precise and accurate definition, but as I understand it, basically, 
PMS is limited to defining the format of ebuilds and the files necessary 
to support them and package-manager interoperability in-tree, including 
eclasses, profiles, EAPIs, etc.  It does NOT cover individual PM 
configuration, etc.

AFAIK, the metadata cache format isn't actually covered by PMS either, as 
to some extent that's considered private to the PM implementation (and in 
the absolute, it could almost certainly be improved upon by individual 
PMs), altho in practice, particularly since it's shipped with the tree, 
that too must be standardized at least for any PM that doesn't wish to 
force regeneration of all that data into its own format at every update, 
when it's already shipped with the tree.

As it became clear a fully compatible implementation simply wasn't 
happening, sets ended up in this gap as well.  Like the metadata cache, 
from a certain viewpoint they're arguably a private implementation 
detail, not subject to interoperability standardization, and I guess this 
is how things were finally finessed.  Of course in practice it would be 
very nice to have inter-operable compatibility, but for various probably 
both technical and political reasons, it simply hasn't happened, and I 
guess the decision finally was to quit letting the impossibly perfect be 
the enemy of the better-than-what-we-had.

So while portage-style sets are now (it seems, this is the first I read 
of it) allowed in-tree, from the PMS angle they remain outside the spec 
as a PM private implementation detail, and thus cannot fully take the 
place of meta-packages.

But as it turns out, meta-packages end up being more flexible and 
powerful anyway, as (AFAIK, maybe it was added and I missed it) at least 
portage's sets implementation doesn't have a parallel to a meta-package's 
USE flags, possible slot deps, etc.

OTOH, as I've found out here, sets are GREAT for users as a method of 
subdividing and organizing the formerly monolithic world file.  Several 
years ago now, as it happens when I was working on setting up my netbook 
and needed to organize my existing world list into categories to better 
decide what bits of it I wanted on the netbook and what bits not, I split 
my world file into about a dozen sets by category (using the kde sets as 
my starting point and pattern), with each former world-file package atom 
placed into one set or another, and all the sets in turn listed in my 
world_sets file.

That makes managing my world list FAR easier, as everything's now 
categorized. =:^)

Posting a set file then becomes a simple way of sharing a list of what 
packages I have installed in a particular category (which may or may not 
correspond exactly to a tree-package category, in the kde overlay several 
kde sets compose the larger kde-base package category, for instance).

Another practical use (extended from the above) I've discovered by actual 
usage of the kde sets, is that I copy and rename the kde sets to /etc/
portage/sets/, and list my renamed versions in my world_sets file.  Then 
I #-comment out individual packages I don't need (as well as libraries, 
so they're just pulled in as deps), while leaving the line otherwise 
intact in the file.

Then every six months when the upstream kde feature release comes out, I 
diff my custom set against the latest kde overlay set, and any new/
renamed/removed packages in the overlay set become immediately apparent, 
so I can update my custom set accordingly.

This sort of usage is perfect for sets, allowing me to coordinate with 
gentoo/kde's package list (in the form of a set) as they adjust to match 
upstream, without having to screw with USE flags and version-deps, etc, 
at the same time.  The package list is managed separately from the USE 
flags and version deps, making things much simpler than they'd be if I 
had to manage them all together, as I would (to some extent or another) 
were I using the metapackages instead.


So to me anyway, sets fulfill a similar but not identical role to meta-
packages, with each having slightly different features and best usages, 
and while one could be made to stand in for the other, the fit wouldn't 
be perfect, and it's best if each one is used for the purpose it fits 
best.  Metapackages remain best where USE flags or slot/version deps need 
to be expressed, while sets are best for simply constructing a list of 
packages to be installed using separately configured USE flags and 
version deps, or shared or compared with another such list.

Given that distinction, the view that sets, as a simple list of packages, 
really are a PM implementation detail, does seem at least arguable.  
Having them inter-operable between PMs would definitely be useful, but 
it's definitely not mandatory, and in fact, given that doesn't appear to 
be doable in anything approaching a practical timeframe, forcing to wait 
until that occurred would indeed seem to be letting the impossible 
perfect be the enemy of the good.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman