[gentoo-user-de] Re: emerge --resume: Läuft nicht so wie ich erwarte

[Thilo Six]

Ich leite dir hier mal eine Mail von der englischen Liste weiter,
die genau dieses Thema behandelt (ziemlich ausführlich behandelt).

viel spaß, Thilo


Betreff: Picking up emerge in middle

 >>>> Hmmm. Good question. I might be talking out my ass here, but I swear
 >>>> that whenever I CTRL+C and come back later, the emerge just starts 
again
 >>>> and jams through all the parts it already did.
 >>>>
 >
 >> I made this experiment:
 >> - during "emerge file" I pressed CTRL-C - made a list of object files in
 >> "work" directory - started "emerge --resume" and stoped it just after
 >> configure finished - checked the list of object files again - they were
 >> lost
 >>
 >> Resume:
 >> "emerge --resume" starts emerge process from the beginning, but new
 >> compile process should be faster due to ccache (if used).
 >>
 >> Nevertheless there is a possibility to continue with broken emerge using
 >> "ebuild" command.
 >>
 >>  From "man ebuild":
 >> Normally, to merge an ebuild, you need to fetch, unpack, compile, 
install
 >> and qmerge.
 >>
 >> So, except of running "emerge --resume" it's possible to run: "ebuild
 >> /path/to/pkg.ebuild  compile install qmerge" which should be the fastest
 >> and very helpful if some hand corrections in source code are necessary
 >> after standard emerge fail ... i.e. due to missing '"' in some packages
 >> (discussed here).


You have most or all the pieces, but don't have them laid out quite as
nicely as I prefer, so here's what I've found.

1 Emerge resume isn't all that useful.

Emerge resume basically restarts a long hairy emerge from a point just
after the last successful ebuild merge.  Thus, it's not really of much
use under normal circumstances, over simply issuing the command again,
since  previously emerged packages wouldn't normally be remerged anyway.
It could however be useful for those that like to set a bunch of shell
variables on the command line before invoking emerge, or otherwise have a
huge amount of typing to redo, if the system crashed or something and
therefore the command isn't available in the shell history.

2 CCache is rather MORE useful, in resumption situations.

If ccache is enabled, David Vincent's observation about emerge just
"jamming thru" compilation steps that have already been completed will be
accurate.  I expect that is what he was seeing.  The effect here is that
portage/emerge does everything over again, but ccache steps in and
shortcuts gcc so the compiles don't actually have to be done over again.
Ccache simply substitutes the cached output (complete with warnings) and
files as if gcc was actually doing it, but faster.

3 Ebuild is even MORE useful, in resumption situations.

Unlike emerge, which automatically deletes the work dirs and starts over,
ebuild will (unless told otherwise) look to see if the unpack step was
successful, and skip it if so.  Further, if by chance the compile step was
ALSO successful, it will skip that also, moving on to the install step
(note the list of the steps from man ebuild in the post I quoted above).

However, since those steps (well, except qmerge, which just merges what's
in the "fake" install dirs to the live system) each skip previous steps
completed successfully, that "compile" step you mentioned above shouldn't
be necessary to specify directly.  Simply ebuild (path to ebuild) install
qmerge should work.

Further, the "merge" action of ebuild combines ALL the above steps, again,
skipping unpack and compile if they've already been done.  Thus, all
one REALLY needs to do is "ebuild /path/to/package.ebuild merge", and it
will verify the unpack step, then pick up with the compile or install step
as necessary and go from there.

**IMPORTANT**  Ebuild does NOT resolve dependencies like emerge does.
Thus, one only uses ebuild on the partially completed ebuild that was
currently being installed when the interruption occurred, if there
happened to be dependencies to emerge before the main package, and the
interruption occurred while emerging one of them, before the main
package emerge began.  Of course, this also implies that the metapackages
"system" and "world" cannot be directly used with ebuild, altho it can
naturally complete the merge of a package merge interrupted as PART of an
"emerge system" or "emerge world".  Just remember that ebuild only works
on one package at a time, and doesn't do dependencies.

4 A bare "make" can be useful as well.

With ebuild, the normal ./configure and make (compile) steps, plus
possible additional tasks specific to the individual ebuild package, are
all one step.  Thus, while it skips the unpack and the invoked make will
skip anything already successfully made (compiled) in the ebuild compile
step, it wills still rerun thru the ./configure testing step each time it
is invoked.

If one expects they might interrupt the compile multiple times (or if one
is running a system unstable enough that it might crash during the compile
multiple times), and one is SURE the ./configure step has already been
completed, one can (with most ebuilds, some work a bit differently
and this won't work, tho it won't harm anything either) actually enter the
work dir (normally /var/tmp/portage/packagename/work/packagename), and
invoke make from there, to do the actual compiling, skipping the
./configure (testing) step that's already been done.  This works because
(again, on most packages) all the fancy --configure-with use flag type
stuff, as well as the CFLAGS type stuff, gets dynamically bound into the
configure script results when that runs, so a simple "make" in the root
source dir already knows what you want compiled and with what flags, based
on that configure information.

The caveat here is that make works at a lower level than ebuild, and thus
has no knowledge of ebuild or how it works, so it can't tell ebuild that
the compile step is finished.  Thus, even after make has successfully
completed, if you invoked it directly, you must still go run ebuild merge
again, and it will still go thru the configure (as it always does, even
when the configure that's there is perfectly usable, because it's all part
of the same compile step), and then skip real fast thru the main make
steps, before marking the compile finished and continuing with the install
and qmerge steps.

The upshot of all this is that if there's going to be only the one
interruption, it does little good to go run make directly, since you have
to run ebuild merge anyway, and it'll do the same thing.  However, if you
may be interrupting the process a number of times, because running the
make directly doesn't rerun the ./configure while ebuild merge will,
you'll save time by invoking the make directly.

The one other factor worth mentioning here is that depending on the
package and sometimes on the use flags, the ebuild compile step will do
more than just a configure and make.  the docs USE flag is a perfect
example, as in many ebuilds, it will cause the ebuild compile step to do
an additional "make docs" command.  Of course, one can always do the
additional steps manually as well, if desired, but that involves
checking the ebuild to see what it actually DOES do, and it's seldom worth
the trouble, since the main make step is almost always the "big" one,
taking the most time and therefore most likely to be interrupted.
Checking the ebuild for the additional steps is likely to take more time
than running the configure an extra time, if you DO interrupt the
process during the extra steps, so it's just not worth it.  However, I do
mention it so folks that take my suggestion and run the bare make, don't
get confused if the ebuild merge then goes and does some MORE make stuff
as well.

...

That's the main stuff, but there are still a couple lose ends to wrap up.

Why all the separate fetch/unpack/compile/install/qmerge steps if a simple
merge automatically checks to see what's done and goes from there?
Simple.  The ebuild command was designed to allow on-site ebuild tweaking,
and the interrupt/resume functionality it provides is more of a side
effect than the reason the tool is there.  That's probably also why the
configure and make steps aren't split out -- it'd serve no useful end
toward the target purpose of allowing on-site installer admin tweaking.

Put a bit more concretely, consider the admin that has a special patch
they want to (or have been told to by upstream developers) try.  While
this is always possible by simply bypassing the portage system and doing a
manual install from tarball, then emerge-injecting the package into
portage's database if desired, ebuild allows just this sort of tweaking at
each step in the process.  One can fetch and unpack an ebuild, then apply
their own patch before running  ebuild merge to complete the process
automatically.  Or, one can ebuild compile and let portage handle that
automatically, then examine the results before finishing up the install
and qmerge steps automatically with merge.  Or, one can do everything up
thru the (fake) install automatically, then stop and examine what files
are in the fake installation before actually doing the qmerge step.

Of course, this also makes troubleshooting an ebuild far easier as well,
because each step ends at a critical point, where the results to that
point can be checked and a problem pinpointed if it has occurred.  If an
ebuild is for some reason going thru the motions and installing all the
docs but failing to install a particular binary, as happened to me once,
it's easy to see if it's in the fake install location before the qmerge to
the live system, and if it's not, to trace back and see if the compile
process actually created the file or if the install simply didn't move it
from the work dir to the fake install dir.  In my case, it was a problem
during make.  I figured out what was going wrong by doing the make
manually until I was able to get the expected file made, then I ran ebuild
merge and let it finish the process for me.  (The problem was a bad path
on my system or some such, so the necessary utility didn't get invoked,
and the necessary file didn't get made, IIRC, thus, a pbcak, problem
between chair and keyboard, user error, not something to file a bug about.
However, without ebuild's troubleshooting ability, I would have had a much
more difficult time figuring out the problem!)

One more item to wrap up.  Check out the noclean and keepwork FEATURES
available to portage, in the make.conf manpage.  I've not tried these, but
by my read, it looks like they may allow an emerge to pick up where it
left off rather than erasing and starting over, just like ebuild does.
The tradeoff of course would be having to manually clean out
/var/tmp/portage (or your non-default portage tempdir location) far more
often, as all those extra packages that are normally cleaned up to a
minimal stub would still be there in full space hogging glory.

That should about cover it!


--
gentoo-user-de@g.o mailing list