[gentoo-dev] Re: Please port your packages to Python 3.8

[Martin Vaeth]

Michael Orlitzky <mjo@g.o> wrote:
>
> Who generates the metadata when I `git pull`?

For the gentoo repository, it is in general some gentoo server
which then pushes the calculated metadata to the repository which
you pull as a user.
It is *possible* to use the "plain" repository, but you have to
set up quite a bit for that (updating metadata is only one of
several steps which you have to do manually in that case).

A collection of scripts which do the missing steps is
https://github.com/vaeth/portage-postsyncd-mv
though I do not know whether they still work. Indeed (as
probably most users) I am using since years one of the
repositories with generated metadata already.

> Or in overlays?

There are overlays which provide metadata, and there are overlays
where you have to do it on your own with egencache. Overlays which
use eclasses from the gentoo repository usually do the latter,
since otherwise the metadata is soon out-of-date.

However, for most overlays, egencache takes far less than a minute,
so for overlays, this is not really an issue. For the gentoo repository,
the time of a full metadata generation is considerable. As mentioned,
checksums and timestamps are used to minimize the amount, though this
does not work for eclass changes (see below).

> but if you record that wasted time in a different "metadata generation"
> ledger, then it looks like I've saved a bunch of time because the "bash
> sourcing" ledger reads zero.

No. You need correct metadata after every syncing of the repository.
Either the gentoo server or your machine has to do it.
This is independent of whether the PM "prefers" the installed or
the repositories' metadata. Excluding installed packages from metadata
updating would not be sane and would not safe much time (since the vast
majority of ebuilds are not installed, anyway).

> Even if I believe in a metadata angel and if we pretend that the PMS
> requires the metadata to be there, then rebuilding whenever metadata
> changes is still not 100% correct (as you point out), because it often
> rebuilds pointlessly. But that's getting into a harder problem.

Yes, usually the metadata rebuilds due to eclass changes are pointless
(except in the few cases where the eclass change is related with the
metadata).

I remember when I used to sync from the "plain" repository and rebuilt
the metadata on my system, that the syncing (i.e. metadata regeneration)
costed 10-30 minutes whenever one of the basic eclasses (which are
sourced by almost every ebuild) had a trivial change. Probably,
meanwhile machines are slightly faster and there are less such "basic"
eclasses needed in newer EAPIs, but it will still need a considerable
time.

> The recompilation isn't always pointless. In the present scenario, the
> rebuild is what installs the python-3.8 copy of the program.

No. If users use the defaults for PYTHON_{,SINGLE}_TARGET, the
rebuild has absolutely no effect except for changing some metadata
in /var/db/pkg (and some file timestamps).

> I'm not arguing that this is the best way to do things, but that it's
> the only way to do them given what's in the PMS today.

Of course. That was the decision made some years ago.

> maybe it should pay a few people to sit around and think about
> this for a week.

There was such a debate some years ago, before the decision mentioned
above was made.
It was a hefty discussion, but there were strong proponents of pointless
recompilation vs. improvement of the dependency handling (for both
sides, there are much more arguments which I will not repeat now).
The discussion might have turned if I would have found the time to
implement the necessary change in portage, but neither then nor now
I have the time to do so. (To be honest: Maybe I had the time, but I
dislike python too much.) Nevertheless, I do not think that it was a
good decision.
I am not posting this to re-roll the above discussion again.
But your posting shows that apparently not everybody knows the reasons
why things are the way they are now.