Re: [gentoo-dev] Re: [gentoo-dev-announce] January 2010 meeting date

[Thomas Sachau]

On 12/26/2009 11:25 AM, Fabian Groffen wrote:
> Hi Thomas,
>
> On 25-12-2009 15:00:36 +0100, Thomas Sachau wrote:
>> On 12/25/2009 06:10 AM, Denis Dupeyron wrote:
>>> Sorry, I forgot to send an email explaining what happened on the
>>> council alias as promised. The consensus was that the project wasn't
>>> mature enough to go ahead. Also more generally the council's job isn't
>>> discussing but deciding, approving, etc... Discussing is what should
>>> happen on mailing lists.
>>
>> Since i see noone arguing against adding the multilib features to
>> current testing branch of portage, the discussion part already seems
>> to be done. so a simple approval is ok, drop the discussion request.
>
> This may be a wasted effort, but I tried figuring out what you do in
> your portage branch going by your earlier discussions.  The idea I got
> is the following:
>
> Triggered by some mechanism (maybe unconditional), you just run each
> ebuild function executed by Portage multiple times, that is, for each
> ABI that you grab from somewhere. e.g:
> src_unpack() {
> 	for abi in ${EABIS} ; do
> 		mkdir ${WORKDIR}-${abi}
> 		cd ${WORKDIR}-${abi}
> 		unpack ${A}
> 	done
> }

Currently it is something like this for every phase:

src_unpack() {
	for abi in ${MULTILIB_ABIS}; do
		set_env $abi
		<do the usual stuff as normal>
	done
}

The ebuild phase does only see the normal WORKDIR, all work with it is done inside portage and
outside the phases, so every package doing things inside WORKDIR just works without additional
changes. The install part contains some additional helpers for headers and binaries.

But i currently try to change it as suggested by vapier:

for abi in ${MULTILIB_ABIS}; do
	set_env $abi
	pkg_setup
	src_{unpack,compile,install}
	mv ${D} ${D}.$abi
done
merge ${D}.$abi into ${D}
<continue as usual>

This way you can set ABI-dependent vars in pkg_setup (previous implementation requires to set it in
every phase, if it was no preserved var), there is nothing in ${D} during src_install, so you can do
a mv again (like some ebuilds or eclasses do or did) and, since the DEFAULT_ABI is run first, i can
detect, if there is actually the need to run everything again for other ABIs.
But i currently cannot say when i am finished with it.

>
> During configure you inject -m{32,64} in CFLAGS depending on your ABI.
> This gives your multilib-compiler the right hint to do the thing you
> want.

Right

>
> This is about what I understood.  Now here I have some questions that
> may or may not be relevant.
>
> What triggers a multilib build?  Is it unconditional, or can it be
> turned on/off per package?  How does Portage resolve/verify that a
> library is built for the right ABI in that case?

Currently multilib-portage does add a USE flag called "lib32". If you enable it, you will get the
cross-compile, else just the normal install. In addition this flag is internally used like an EAPI-2
usedep, so it will require the dependencies to be built for all ABIs too.

>
> Unpacking sources many times feels like a terrible waste to me,
> especially for things like GCC.  If we would just start building outside
> of the workdir (sources) into a separate builddir, wouldn't that just
> be much cleaner and a nice EAPI feature?

That might be an extra step, once the basic implementation works, but you will have to adjust some
things, since e.g. cmake-utils eclass does already something like that, maybe others do it too, so
you would have to change those ebuilds/eclasses or add exceptions or extra rules to portage for those.
Some packages like gcc or glibc already do this multilib-stuff internally with the multilib USE
flag, so you currently wont get any better experience for them.

>
> Since you make each compilation multiple times, you also obtain a fully
> identical installation of the same package.  How do you deal with that?
> Do you have /usr/bin{64,32} directories for example too?  If you only
> keep libs (found by a scanelf scan or something), how do you know what's
> relevant.  Alternatively, if you build the full application anyway,
> isn't it a waste to throw away the result?  You could see multilib also
> as two (unrelated) trees, such as e.g. a Gentoo Prefix installation.
> A nasty one: how to deal with libs that actually contain hardcoded paths
> to configuration from e.g. /etc or /var in your implementation?

Currently i only work and test with amd64-ARCH, so with x86 and amd86 ABI. For those, the lib-part
is easy since the crosscompile does install the libs into /usr/lib32 while the 64bit ones go into
/usr/lib64. The headers for both ABIs are diffed and different ones are preserved, the rest is
isntalled as usual. For binaries, normally only the one for the DEFAULT_ABI, so in this case the
64bit one, will be preserved. But you can tell multilib-portage to preserve the 32bit binaries. In
that case, the binaries will be called $binary-$ABI and a symlink $binary to a wrapper created,
which calls the real binary depending on the current ABI.

>
> You chose to inject -m{32,64} in CFLAGS.  Suppose you set CC to "gcc
> -m{32,64}" or even "x86_64-..." or "i686-..." you could do some
> cross-target stuff, I think.  I say so in the light of Darwin systems
> which are capable of using Mach-O FAT objects.  Such objects can contain
> multiple architectures.  This idea is available for as FATelf also, and
> e.g. included in the zen-sources.  On such system you ultimately want to
> handle the multilib hel^Wproblem by avoiding the different paths but
> instead generate that single unified tree that contains all your ABIs in
> each (FAT) file.  Is your approach flexible enough to lipo two or more
> trees together in one after the two images are installed?

My currently planned changes do install into different target dirs, so if those target dirs do
contain all you need, there should be no problem to add some additional lines during merge.

>>> Before you can bring that to the council we
>>> need to see an as-much-as-possible finalized solution with any of the
>>> following if applicable: portage branch with an implementation that
>>> people can try, documentation, PMS patch, devmanual patches, and a
>>> team.
>>
>> Did you actually read my lines? I did NOT request an ACK to add it to
>> PMS and next EAPI with a complete spec. zmedico also has no problem
>> with having a look and adding it, but since he was once forced to
>> remove an added feature, he now wants a council-ok before adding and
>> improving it in testing branch of main tree portage.
>
>>From my experience they just want to get some grip on the issue.  A
> formal description helps sometimes.

As i already said: My implementation is not final nor do i request some PMS changes or EAPI bump for
it. I simply want some more help and feedback by getting it in the 2.2_rc* branch of portage and
zmedico just wants an ok from council, so that they wont force him to remove it again.

Once the implementation is final, there should be more time to create the SPEC, PMS patches and
similar for a final approval by council and inclusion in PMS/some EAPI.

>>> Look at what happened with prefix. They wanted
>>> the council to approve it immediately or else... We didn't cede to
>>> pressure and worked with them to make it good enough for approval.
>>
>> Something like "I/We want <x>,<y>,<z> or you wont get an approval" is
>> no support and no "work with them". So if you really would like to see
>> it in, actually help with patches, SPEC writing, discussion and code
>> writing. Else i request an approval for getting some additional help
>> instead of just shooting it down.
>
> Pfff, I guess like you, Thomas, we got a bit impatient.  Our experience
> is that once you give the information to the council in the right
> format, they seem to be much more focussed.

But i currently dont have "the information", i only want a "we wont force the portage team to remove
the multilib code from portage-2.2_rc*".

>>> Right now I don't hear anybody arguing about prefix going forward. And
>>> that's exactly what I want for your project, i.e. helping you making
>>> it better instead of it fading and failing in the (not so) long run.
>>
>> prefix is no one-man-team and the actual amount of people, who can and
>> are willing to work on portage code is limited, so which other way do
>> you have to improve it as requested?
>
> Prefix has been more or less a one-man-team for a long long time, but
> then, the project exists for about 5 years now.  I cannot really advise
> you to go that route, so please try to spec it.

Long term forking is hard, especially when you also work for other time consuming projects and have
some work, where you have no contact with Gentoo or Linux.
But i cannot give out specs, when it is not final. My main interest is more feedback, more comments,
more testing and maybe a bit more help for the code part.

-- 
Thomas Sachau

Gentoo Linux Developer