Re: [gentoo-dev] Re: have portage be quiet by default

[Zac Medico]

On 11/14/2011 03:17 PM, Duncan wrote:
> Zac Medico posted on Mon, 14 Nov 2011 07:22:19 -0800 as excerpted:
>
>> On 11/14/2011 12:47 AM, Dirkjan Ochtman wrote:
>>> On Mon, Nov 14, 2011 at 02:59, Zac Medico <zmedico@g.o> wrote:
>>>> Well, it's much easier to gather interest and get feedback if we
>>>> deploy the change and ask questions later.
>>>
>>> I like the new output, but find it kind of annoying that there's very
>>> little feedback on how far the progress is within a single job. Perhaps
>>> we could show the currently executing ebuild phase in order to give a
>>> little more feedback? Maybe most packages are completely dominated by
>>> src_compile(), but for smaller packages it would be helpful, IMO.
>>
>> I think that would be an interesting option. I imagine that it would be
>> too much information for many people, so I don't think that we'd want to
>> enable it by default.
>
> Thanks for thinking "out of the box". =:^)
>
> This is one thing that I've found frustrating with parallel emerging,
> myself.
>
> The first thing I discovered were that some phases (at least the merge
> phase, and I believe install) aren't counted, so the numbers don't
> entirely add up (complete + running + not-started + failed <> total).
> Thus, some packages appear to be simply sitting there doing nothing.

This behavior is discussed in the "When packages are built in parallel
with the --jobs option, why aren't some packages installed immediately
after they have finished building? They are installed only after a long
delay." section of the FAQ [1].

> When a whole bunch of packages are in that state, it appears emerge is
> doing nothing but consuming cycles and real-time, for no visible reason
> at all.

I haven't noticed anything like that. You can send a SIGUSR1 signal to
the emerge process and that will cause it drop to a pdb shell so you can
poke around and see what's happening.

> Since for a number of packages (I seem to notice it most on kde packages,
> but perhaps it's all C++ or most CMAKE or some such, plus of course
> primarily data packages that don't have a significant build phase but
> install many or large files) the install phase can be as intense as the
> build phase if not more so, so if some phases aren't counted in
> "running", then I'd definitely prefer they be counted /somewhere/.

The src_install phase is counted along with all of the earlier phases.

> Of course, listing the number of packages in each phase (which in single-
> mode would by definition list the phase the single package is in) would
> cure the above issue at the same time.
>
> Another problem, but one I'm not sure how to fix (tho the interactive
> display would offer opportunities here), is that it'd be nice to get a
> list of completed packages, in-process packages, and still not started
> packages.  That doesn't so easily fit in a neat summary, but as noted,
> the interactive proposal introduces quite some opportunity, here.
>
> Alternatively, since often the real info desired is the status of a
> particular package, it'd be useful to have either an interactive command
> or a simple separate querying command, that can be run to query the
> status of a particular package.
>
> *** Which brings up an alternative to the whole interactive emerges idea
> as well -- what about a separate command, say "emerging", that when run,
> would simply output a bunch of detail on any pending emerges?
>
> This would certainly offer a safer initial implementation, as well, since
> it's less likely to break the current setup due to bugs, etc.  Another
> noted benefit is that it leaves the existing defaults alone, so it's not
> going to be threatening anyone's sacred cows (or scripted assumptions) in
> terms of how portage has always behaved. =:^)
>
>
> So, perhaps improve the current "quiet" display marginally, either
> including all phases in "running" or adding another listing so the
> numbers add up, but more importantly...
>
> *** Add a new "emerging" (name up for bikeshedding) command, that
> displays a nice multiline summary, perhaps something like this (the
> second emerge command was for a failure in the first, with a second
> attempt made before completion of the main emerge @world has completed):
>
> Current emerge status:
>
> 2 emerge command(s) running.
> 7 emerge jobs outstanding.
>
> Commands:
>
> ** emerge --upgrade --deep --newuse --keep-going @world
>
> Status:
>
> 123 of 257 jobs complete, 1 failed, 6 running, 120 not yet started,
> 7 dependencies removed due to failures.
>
> Running (6):
>
> 1 pkgsetup (cat-egory/package-ver)
>
> 1 configure (cat-egory/package-ver, cat-egory/package-ver)
>
> 2 build (cat-egory/package-ver, cat-egory/package-ver)
>
> 1 install (cat-egory/package-ver)
>
> 1 merge (cat-egory/package-ver)
>
> Failed (1):
>
> dev-libs/boost-1.47.0-r1
>
> Complete (123):
>
> <list		in		nice		columns>
> ...
>
> Not started (120):
>
> <list 		in		nice		columns>
> ...
>
> Removed dependencies due to failure (7):
>
> <list		in		nice		columns>
> ...
>
>
> ** emerge -u1 boost
>
> Status:
>
> 0 of 1 jobs complete, 1 running.
>
> Running (1):
>
> 1 build (dev-libs/boost-1.47.0-r1)
> Obviously the first implementation might not have all the information
> above, and there might have been some nice information I missed and other
> information that could be displayed better, but it's a very cool idea
> even if I /do/ say so myself. =:^)
>
> Note that zeroed-out listings aren't displayed, so the the --oneshot
> doesn't list failed, not yet started, etc, and inactive phases are also
> not displayed.
>
> Also note how easy this sort of detailed display would make it to retry
> failed jobs and the resulting removed dependencies, once the failure has
> been resolved.
>
> "emerging" could have a "repeat" mode as well as one-shot, with a polling-
> delay parameter set to something sane like 30 or 60 seconds by default.
> That way, I wouldn't have to feed it to "watch" =:^)

I'm not sure how many people would use an interface like that in
practice, but I wouldn't be opposed to adding support for something like
that. Since I'm not really interested in using an interface like that
myself, so I don't feel inspired to implement it myself.

> Because this is a separate command, worries about TMI/TLI should be
> eliminated.  Additionally, it should go some way toward easing the whole
> quiet/noisy debate as well, since there's now another command available
> with an intermediate level of information.
>
> Something like this:
>
> edisplaylog
> If we then throw in one more tool, I'll call it edisplaylog for lack of a
> better name, that reads make.conf to find the elog dir, and can
> automatically pick out the latest log for a package, thus eliminating the
> trouble of doing it manually, that should help eliminate another
> objection to quiet-by-default.  So...
>
> edisplaylog boost
>
> ... would find the last elog for boost and display it.
>
> To make things easy for the user, simply ...
>
> edisplaylog
>
> ... could be made to filetime search and display the last active log in
> tail -f mode.  If after say a couple seconds of no activity, it re-polled
> the log dir and switched to displaying a different log (have it list the
> log it's displaying at the top) if it was newer, that could pretty much
> replace "noisy" mode entirely.  =:^)
>
> Obviously running simply "edisplaylog" by itself during a parallel-jobs
> emerge would be nearly as confusing as portage not automatically going
> into quiet mode for parallel emerging would be, tho not quite since it
> would display the logfile name at the top, but "edisplaylog <pkg>" would
> still be very useful, ESPECIALLY for those using --keep-going, so they
> could track down and fix whatever failed a build before the parallel
> emerge spit out the results at the end.
>
> Talking about which... an "edisplaylog failed" mode, which would
> automatically find the last failure and displayed the log for it, could
> be very helpful as well. =:^)

That sounds pretty handy.

> Obviously all these ideas entail a lot of coding, and I realize it's easy
> to sit here and make requests without doing the coding, but implementing
> them as separate commands first and incremental implementation should
> help a lot, and I hadn't seen anything like this proposed yet, so...
>
> I'll close with a thanks to Zac and everyone else who has already made
> portage the great tool it is today, because without them, we'd obviously
> not be having this discussion in the first place. =:^)
>

[1] http://www.gentoo.org/proj/en/portage/doc/faq.xml#doc_chap1_sect9
-- 
Thanks,
Zac