Re: [gentoo-user] Tailing compressed build logs

[Bryan Gardiner]

Hi,

On Tue, 07 Mar 2023 09:01:34 +0000
Michael <confabulate@××××××××.com> wrote:

> On Tuesday, 7 March 2023 07:52:01 GMT Mickaël Bucas wrote:
> > Le mar. 7 mars 2023 à 05:36, Bryan Gardiner <bog@××××××.net> a
> > écrit :
> > > Hi folks,
> > >
> > > How can I follow Portage's compressed build logs in real time as
> > > they are generated?
> > >
> > > ...
> > >
> > >     tail -c +1 -f build.log.gz | od -t x1
> > >
> > > but the following hangs with no output:
> > >     tail -c +1 -f build.log.gz | gunzip
> >
> > Reading the man page, "zless" is just a wrapper around "less".

Neat, zless is a lot simpler than I thought.

> > So it should support the same options including typing "F" at the
> > end of a file to keep trying to read when the end of file is
> > reached.
> >
> > I made a small test, but it didn't work:
> > # Create a growing file
> > $ yes | nl | gzip > zless-test.gz &
> > # Try to follow at the end
> > $ zless zless-test.gz
> >
> > With ">" to go to the end and "F" to continue, I didn't get the
> > expected behavior, it stood still at the point I was viewing.
> > I don't know if it's really a bug or if I made a mistake...
> > (Don't forget to stop the growing file :) )

I suppose it makes sense that a simple "gunzip -c file.gz | less"
doesn't work, assuming zless is equivalent to that.  gunzip sees the
end of the file and exits rather than waiting for additional content.
Then less sees its pipe close.

So I think the "tail -c +1 -f build.log.gz" bit is needed, and appears
to work since od shows data appended every few seconds.

> You could try:
>
> tail -c +1 -f build.log.gz | gunzip | less
>
> I think it should work, but I haven't tried it.

Thanks, but I suspect that less is still at the whim of gunzip here,
and gunzip isn't producing anything.

Maybe gunzip just has a massive output buffer.  If I do something a bit
more complicated:

    (N=0; while true; do
      (( N++ ))
      echo "$N" | gzip
      if (( N % 4000 == 0 )); then sleep 1; fi
    done) | gunzip | od -A d -t x1

then nothing is shown for 15 seconds or so, then od quickly dumps
51952 bytes of hex output, then another long pause before dumping
another 60480 bytes of hex.  If I try this with "pigz -cd" instead of
"gunzip" then output starts flowing after just ~400 bytes, but sadly
that still doesn't work for build.log.gz.

Anyway, I've had another thought, that since portage is flushing lines
to the gzipped log fairly frequently, it's probably not getting the
best compression ratio.  So I went and tested recompressing some logs,
and got a decent improvement:

- nodejs-18.9.1: build.log.gz is 112KB, recompressed it's 68.9KB
  (38.5% savings).

- qtwebengine-5.15.8_p20230112: build.log.gz is 1.44MB, recompressed
  it's 1.01MB (29.9% savings).

- gtk+-3.24.37: build.log.gz is 157KB, recompressed it's 120KB
  (23.6% savings).

So maybe I'm better off abandoning this feature, and setting something
up to compress build logs after the fact.  Then good ol' tail -f will
work just fine.

Thanks, cheers,
Bryan