Re: [gentoo-alt] AIX linking adventure

[Perry Smith]

On Feb 10, 2011, at 3:06 AM, Michael Haubenwallner wrote:

>
> On 02/10/11 01:57, Perry Smith wrote:
>>> On 02/09/11 17:47, Perry Smith wrote:
>>>> Lets start over.  I don't think I'm making any progress but I'll give it one last go.
>>>>                       ***Import File Strings***
>>>> INDEX  PATH                          BASE                MEMBER
>>>> 0      /usr/local/rvm/rubies/ruby-1.9.1-p378/lib:/usr/local/lib/gcc/powerpc-ibm-aix5.3.0.0/4.3.1:/usr/local/lib/gcc/powerpc-ibm-aix5.3.0.0/4.3.1/../../..:/usr/lib:/lib
>>>> 1                                    librtl.a            shr.o
>>>> 2      /usr/lib                      libc.a              shr.o
>>>> 3      /usr/lib                      libpthread.a        shr_comm.o
>>>> 4      /usr/lib                      libpthread.a        shr_xpg5.o
>>>> 5      /usr/local/rvm/rubies/ruby-1.9.1-p378/lib libruby.so
>>> Is this with our without your intervention in ruby's build process and/or toolchain setup?
>>
>> This is after my toolchain changes.
>
> How does this look like without your changes?

I can't recall 100%.  I can go back and do another build.  In particular, I can't recall
who had the problem if it was ruby itself or libruby.so that had the problem.  In general,
there were no "hardcoded" elements and the 0th element was simply:

/usr/local/rvm/rubies/ruby-1.9.1-p378/lib:/usr/lib:/lib

Let me know if that is sufficient.  If not, I can do another build without my mod's.

>
>> The 1st entry is what I was calling "relative" but as you point out, that isn't the right term.
>> "affected by run path search" is too long for me. :-)
>>
>> The 2nd through 5th items are those I was calling "absolute" but really they should be
>> called "not affected by run path search".  Perhaps "fixed" in this case would work?
>> non-LIBPATH-able?
>
> IIRC, "hardcoded" is what libtool uses for that kind of paths,
> but I'm fine with "fixed" too.

Lets use existing terms if they exist.
hardcoded isn't too long.  what is the opposite?  non-hardcoded?

>
>>> How exactly did you touch either the build process or the toolchain setup to "allow" that?
>>
>> I put my ld in /usr/local/aixbin/ld and I put /usr/local/aixbin as the first element of PATH.
>> So my ld gets called instead of the real ld.  My ld monkeys with things.
>
> Which compiler did you use - gcc or xlc?

gcc 4.3.1

> Does xlc search for 'ld' using PATH?

Good question.  I don't know.

> Usually, gcc does not, you'll have to configure it to use "/usr/local/aixbin/ld".

collect2 (from gcc) calls ld.  I'm using my "stock" gcc build and its working for me.

>
>>>> Lets pause here and see if we can come to an agreement on these two items.
>>>> 1) objets have only absolute paths after they are installed
>>>
>>> Still not clear to me:
>>> Do you actually /prefer/ hardcoded paths (without runpath search)? Why?
>>>
>>> Although runpath search is affected by LIBPATH, I'm just fine with that.
>>> This is the outcome of using "-L" and "-l" linker flags together.
>>
>> I thought this was one of your objectives based on the fact that java sets LIBPATH
>> to something weird and it breaks everything.
>
> Nope. The problem with LIBPATH set by java was that there is nothing
> like "soname". Iff my binary had searched for the /file/ "libiconv.so.2"
> (either archive or not) instead of "libiconv.a(libiconv.so.2)", even with
> LIBPATH set to "/usr/lib" it would not have found "/usr/lib/libiconv.a"
> (without "libiconv.so.2" member), but continued along LIBPATH + runpath
> until it found the "libiconv.so.2" file in its usual place.

Hmmm... ok.  I had forgotten that detail.

>
>>> Seems we have to agree on some terms to be used first.
>>
>> Yes.
>>
>> I like "fixed" for entries not affected by runtime search.
>
> "fixed" = "hardcoded" (see above).
>
>> The other tact would be to note that PATH is either empty or not.
>>
>> "A path dependency with no path" or "a dependency with a path" ?
>
> Which "PATH" ("path") do you refer to here that can be either empty or not?

In the dump -H output above, there is a column called PATH.  That PATH.  I apologize for being
so consistently vague.

> There is the "PATH" environment variable, which does not affect runpath search at all.
> There is the "LIBPATH" environment variable, which does.
> There is the encoded "hardcoded path" (="fixed") for one library, not allowing runpath search.
> There is the encoded "runpath", used (after LIBPATH) for any library without a "hardcoded path".
>
>> "binary" instead of "object" sounds better to me.
>
> ok.
>
>>> What else to say:
>>> When libtool knows for a platform that linking libraries results in hardcoded
>>> paths, or there is no such thing like LIBPATH environment variable to temporarily
>>> override the runpath, it actually /does/ "relink" the binaries during installation.
>>
>> I understood that part -- but
>>
>>> OTOH, when the /absolute/path/to/the/library.so found at linktime is hardcoded into
>>> the resulting binary, installing into DESTDIR is broken - which is a prerequisite
>>> for package managers like portage to work properly.
>>
>> Didn't understand this part.  In particular, from the comma to the end ", installing ..."
>
> Ohw, ok - this is the core requirement for proper package managing:
>
> Consider these raw build steps for an usual package, which does provide
> "${prefix}/lib/library.so", and "${prefix}/bin/mybinary" linked against it:
>
> 1. Configure:
>   $ ./configure --prefix="/final"
> 2. Compile:
>   $ make
> 3. Install (to allow for a binary package):
>   $ make install DESTDIR="/image"
>   The important thing here is to /not/ copy anything to "/final" at the
>   build machine, but still the binaries copied to "/image/final" have
>   to be able to run when moved to "/final" by the package manager,
>   eventually on another machine with (ideally) identical setup.

Does the package (need to) work from the /image area or is this
just a temporary place used to create the tar file?  In particular,
what about paths hard coded into the binaries?  (Not the
paths in the binary's header but just paths hard coded in the code
like where to find my lisp files, etc.  I know that *usually* there are
options to change them but there is still some default set at compile
time.

> 4. Create the binary package:
>   $ cd "/image"
>   $ find "final" > "filelist-this-package-does-install"
>   $ tar cfj "binpackage.tar.bz2" "filelist-this-package-does-install" "final"
> 5. optionally ship the binary package to another host with (ideally) identical setup.
> 6. See if the binary package would fit:
>   $ tar xfj "binpackage.tar.bz2" "filelist-this-package-does-install"
>   $ for f in $(<"filelist-this-package-does-install")
>     do
>       if [[ -e /${f} ]]
>       then
>         do-we-update "${f}" && continue
>         echo "Installing this package conflicts with existing '/${f}'"
>       fi
>     done
> 7. Merge the binary package to the live filesystem:
>   $ cd "/"
>   $ tar xfj "binpackage.tar.bz2" "final"
> 8. Use the package:
>   $ "/final/bin/mybinary"
>
> During step 3, the shared library gets installed as "/staging/final/lib/library.so".
> Relinking "mybinary", the linker has to read that file, while encoding "/final/lib"
> (without "/staging") into "mybinary" as the search path for "library.so".

Shoot... you lost me.  I think maybe /staging is the same as /image above?

>
> Usually that is done this way, encoding the final "runpath":
> $ ld -o "/staging/final/bin/mybinary" -L "/staging/final/lib" -lrary -blibpath:"/final/lib"
>
> Because this won't work as expected, as the "hardcoded" path would contain "/staging":
> $ ld -o "/staging/final/bin/mybinary" "/staging/final/lib/library.so"
>
> Except... when an "Import File" is found at "library.so", referring to "#! /final/lib/...".
>
> However, when "library.so" is a /standalone/ Import File,
> it has to refer to something like "#! /final/lib/library.so.0",
> so the loader actually can load a shared object rather than an Import File.
>
> When "library.so" is an archive file /containing/ both the Import File "shr.imp" and the shared object "shr.o",
> the Import File has to refer to "#! /final/lib/library.so(shr.o)".
>
> For completion, "library.so.0" is called the "soname" in ELF world,
> but that's a different (although related) topic.

Ok.  I have more thoughts on this too.  Lets not go there just yet.  I'm still
lagging just a little.

One question: with each of your experiments, do you dump out the
binary's "header" with dump -H?  The reason I ask is I would expect
linking with an import file basically just put that piece, the #!<whatever>
into the binary's header at link time and then what happens at run
time is the same as if you had linked against a library.

I remember toying with the import file inside the archive (and getting
frustrated at the lack of documentation) but I can't remember what
I found.