Re: [gentoo-portage-dev] Speeding up Tree Verification

[Pacho Ramos]

El mar, 30-06-2020 a las 08:20 +0200, Fabian Groffen escribió:
> Hi,
>
> On 29-06-2020 21:13:43 -0500, Sid Spry wrote:
> > Hello,
> >
> > I have some runnable pseudocode outlining a faster tree verification
> > algorithm.
> > Before I create patches I'd like to see if there is any guidance on making
> > the
> > changes as unobtrusive as possible. If the radical change in algorithm is
> > acceptable I can work on adding the changes.
> >
> > Instead of composing any kind of structured data out of the portage tree my
> > algorithm just lists all files and then optionally batches them out to
> > threads.
> > There is a noticeable speedup by eliding the tree traversal operations which
> > can be seen when running the algorithm with a single thread and comparing it
> > to
> > the current algorithm in gemato (which should still be discussed here?).
>
> I remember something that gemato used to use multiple threads, but
> because it totally saturated disk-IO, it was brought back to a single
> thread.  People were complaining about unusable systems.
>
> In any case, can you share your performance results?  What speedup did
> you see, on warm and hot FS caches?  Which type of disk do you use?
>
> You could compare against qmanifest, which uses OpenMP-based
> paralllelism while verifying the tree.  On SSDs this does help.
>
> Thanks,
> Fabian

I am talking only based on my own experience but, at least on my case, there are
noticeable differences between I/O schedulers. That is something that maybe
could affect that tests... with latest kernels we only have noop, deadline
(default) and bfq... in my case bfq works far better for keeping the system
responsible (on both, rotational and SSDs). But you can test different
combinations to see the one that fits better for you.