Re: [gentoo-portage-dev] DB and binary dependency

[Paul de Vrieze]

On Friday 24 March 2006 15:24, tvali wrote:
> > Unfortunately, your wrong. This only makes sure that you have the
> > right slots to put your squares, triangles and circles in. It does
> > not say that b(int,int) from the first lib actually does the same
> > thing as b(int,int) from the second library.
>
> Sorry i thought about it for a moment, then left out later as broken
> design cases didnt seems so important. Versioning of distinct
> functions would be good, anyway not automatic.

In many cases these things are not broken design. My example is, to make 
it clear. Unfortunately at the moment you do automatic dependencies, you 
MUST be correct between 99.99999% and 100%. These cases are too common to 
disregard. Changing semantics are common enough. Sometimes the old 
semantics are just broken.

> > While the above example is clearly broken design, this does happen
> > enough in actual libraries in way more subtle ways. And that is
> > disregarding the fact that the linux/elf ABI does not include
> > argument lists in symbol linking. As such b(int,char) is
> > indistinguishable from b(int,int). To overcome this C++ uses name
> > mangling which creates names based on the signature.
>
> Ok, assuming broken design it's harder. Anyway, how would binary deps
> solve this? This seems to be humans work in all cases.

We don't look at what packages do. We take a default heuristic, and let 
humans give the correct answer when the default is wrong. 

> Anyway, as .h files have become as well as standard, you may show in
> code as well as in bin, which functions are needed.

Except that the way header files are used is not standard. You know there 
is a reason for the way gcc handles precompiled header files. That is 
because of the way they work. Header files can have unpredictable 
interactions. Including A then B can be really really different from 
including B then A. While in most cases it isn't, it is in too many 
cases.

> Is there any advantage of bin dep over code dep is a question for me
> ..some here suggest that there is, i think that everything can be done
> as well with code deps.

bin dep is not about how to get the dependencies. Binary dependencies 
specify what the dependencies have become after compilation. Binary 
dependencies basically strengthen the runtime dependencies, by taking the 
concrete dependencies used into account.

With binary dependencies comes the need to verify them. This is because 
too many ebuilds do not use the version of a multislotted library that is 
expected. As such verifying the dependencies of a binary package gives 
the opportunity for developers to signal and fix this.

>
> > > I'm actually sure that this all can be calculated up from
> > > sourcecode and bindep would be after that a check if cpu didnt
> > > calculate something wrong :P Another question, how difficult it is
> > > and is it worth the time.
> >
> > Perhaps you should read up your knowledge of the C language. After
> > you found that the C language is a mess, try C++, it makes things
> > worse. After that's finished take a look at solving this problem for
> > ALL languages.
>
> I actually know c++, but dont know c. Anyway, what i mean is that if
> you have .h files, you may make up automatic check if interfaces fit
> each other. I have to think about if c++ is a mess :)

C++ is every bit as much a mess as C as it is a superset. The problem is 
actually partly in the C language and partly in the preprocessor. First 
the preprocessor is fragile, and can depend on arbitrary command line 
options (Defines). The language problem is that C/C++ has no notion of 
modules. file a.c can define a function as "int a(char)" and b.c can have 
its prototype as "int a(wchar)". The compiler will not complain about 
this, AND with the right defines it IS actually the same.

The fundamental problem is that the compiler only knows about symbols that 
may be imported from other object files. It does not look at the actual 
object to look at the prototypes of the symbols defined there (nor do .o 
files outputted by gcc contain this information). Instead header files 
can be used to share this information. Often they do this in a broken way 
and errors similar to the above exist.

>
> > Automatic source analysis for dependency calculation is a dead end.
> > Even if you manage to find the proper interfaces (oops, the package
> > had it's own gl.h instead of using the system one), you don't know
> > anything about the semantics of those interfaces. Two things with the
> > same name may very well have very different behaviours.
>
> I think it should be solved with:
> // system's header
> #include <...>
> // own header
> #include "..."

It's not your code, you can't change it. And including two headers which 
define the same symbols can be disastrous.

> Ok -- i agree that there are several problems, which have to be solved
> before taking this topic up again and that they are possibly
> unsolvable in many cases and possibly too much work in many others.

The thing is,

with reasonable effort you are probably going to be able to make something 
that works in 95% of the cases of C and C++ programs. This however is not 
enough to base automatic dependencies on. You need something a lot more 
reliable.

For verification (instead of automatic deps) 95% is however acceptable. 
The problem stays however that many of the packages in the tree are not C 
or C++ based (or worse, mixes). Analysis of binary files is something 
that works on all elf objects, regardless of their language or compiler. 
The results of analysing are roughly the same as the result of source 
analysis. It is "A LOT" easier to do though as there are tools, and you 
don't need to parse at all. It's also faster to do, as the ELF format is 
highly structured. Additionally one could do a fast scan of "#include" 
headers of all installed header files.

These things are all not done yet, and quite cheap to do (both in time for 
running, as in development time). This verification will probably yield a 
better coverage than source analysis will. It is also closer to what we 
are interested in.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@g.o
Homepage: http://www.devrieze.net