[gentoo-user] Re: dynamic deps, wtf are they exactly

[James]

Michael Orlitzky <mjo <at> gentoo.org> writes:


> With dynamic deps, portage will scan (that is, execute) all of the
> ebuilds for installed packages that could affect the dependency graph.
> If any of those ebuilds have changed, portage will use the new
> information rather than the info present when you originally installed
> the package. So if the gtk ebuild (that I installed a month ago) has
> been altered, portage will re-read it on the fly, ignoring what was in
> there a month ago.

Seems reasonable from a  first glance point of view.


> That's nice, because now if you want to install or update something
> else, portage doesn't have to re-execute every ebuild/eclass that could
> possibly affect the new thing -- it only has to check the VDB.

I think that this is what GLEP-64 is all about. Enhancing the functional
utilityh of the VDB with a DAG (Directed Acyclic Graph)?


> But, if I'm the maintainer of nano and I change that ncurses dependency
> (let's say I drop it), then I have to make a revision bump on nano.
> Otherwise, the wrong dependency would stay recorded on everyone's
> system, and portage would happily use the recorded deps.

Basically from my point of view, something like TUP [1] is needed so
that at dependency check time you only list files that need
attention (linking, loading, compiling etc) thus speeding up the
update processes for the Package Manager (portage). I do not study
Palaudis or others.



> I guess at some point there were a bunch of devs who were messing with
> dependencies and not bothering to make revision bumps. This can cause
> users pain, so portage added a new option to ignore the cache and rescan
> every single relevant ebuild/eclass for sneaky dependency changes. This
> ensures that portage has the correct dependencies in its head while it's
> doing resolution, but is much slower.

There is no proper mechanism to accurately track all of these issue,
currently, or did I miss this point?


> And then that mode was made default, which is where we are today. It
> doesn't sound that bad so far -- just a tradeoff between speed and the
> risk of using an incorrect cached dependency.
>
> Buuuuuuutttttt dynamic-deps mode doesn't always kick in. It interacts
> weirdly with subslots and other things. If portage can't find the same
> ebuild to scan, it will find one that "looks like it." If that doesn't
> work, it's supposed to fall back to the cache. Nobody has a flow chart
> of exactly how this works. It's all in the code and there's no
> specification.

It' a difficult subject so half baked measures abound, imho. A full DAG of
the things in the VDB that tracks and retains additional information, is
currently one possible solution. But the DAG only collects the data needed
to develop more robust tools, or at least that is my understanding. I have
read that Ninja, could implement a DAG, but the devs have not made that
commitment yet for Ninja. Neil posted a while back about "CheckInstall"
which addresses some of the problems that are related too. Thesre are not 
package managers but merely "tools" that could be employed to resolve some
of the related issues.


> And, there are other package managers besides portage. When developers
> rely on dynamic deps and skip revision bumps, they're screwing users of
> paludis and pkgcore (and you, now that you have dynamic deps disabled).
> None of the magic is mentioned in the PMS, so you really can't rely on
> it and revbumps are needed regardless.

The Package Management Specification [3] is also intertwined in this issue.
It's a very complicated issue and a problem everywhere you have complex
codes and large numbers of codes and packages that define a system. [[4] an
excellent read].  Please, folks with deeper understanding, do write as much
as you can. An updated gentoo dev blog post, at least framing the issues,
would be keen background for everyone, imho.


It becomes even more complicated  when you look at clusters, the resources
and the frameworks to solve problems, including Continuous Integration,
compiling across many languages and other goals of the cluster. I am far,
far away from possessing a sufficiently deep understanding on these issues.
I just peel back a bit of the onion most days and try to discern the issues
 I encounter:: designing a proper cluster for gentoo  does lead one down the
path of unravelling these aforementioned issues, and other issues, from the
myriad of codes, packages and strategies for distributed computing.


hth,
James

[1] http://gittup.org/tup/

[2] http://asic-linux.com.mx/~izto/checkinstall/

[3] https://wiki.gentoo.org/wiki/Project:Package_Manager_Specification

[4] https://blogs.gentoo.org/blueness/2014/08/