Re: [gentoo-dev] Gentoo Council Reminder for May 28

[Tiziano Müller]

Am Donnerstag, den 28.05.2009, 09:23 +0200 schrieb Patrick Lauer:
> On Thursday 28 May 2009 07:46:36 Tiziano Müller wrote:
>
> > And here is why (I'm only looking at the non-degenerated case with valid
> > metadata, ignoring overlays which some consider a corner case (I don't
> > understand that argument, but that's another thing)):
>
> overlays tend to come without metadata. Just enabling the KDE overlay changed
> the time for "emerge -upNDv world" from ~30 seconds cold cache to ~120
> seconds. Running emerge --metadata gets the performance back to pretty much
> the old levels.
>
> > When the package manager looks at a package, it first reads the
> > package's ebuild directory and gets the mtimes. It does the same for the
> > cache entries and validates the caches (there is more stuff in here,
> > like checking eclasses and so on).
> Eclasses are negligible because you only have to look at them once for the
> whole caclulation. You can cache the mtime for the duration of your operation.
>
> > Then the following happens based on the "solution" we choose:
> > eapi-in-filename: the package manager starts from the highest version
> > with a supported eapi (the others are inexistant with the used glob).
> > For that ebuild it reads the cache entry and decides whether or not it
> > can be used.
> In this case you amusingly do NOT want to cache the eapi in the cache, so you
> can even defer sourcing the ebuild until you actually need the metadata.
by "whether or not it can be used" I meant "keyword-like", surely not
eapi-like since you already know it at that point.

> (You don't want to cache it because you need to check the file mtime anyway,
> and then you read the filename anyway. No need to look for it in another place
> then :) )
> > If not, it proceeds to the next version, if yes, it's done.
> > eapi-in-ebuild: the package manager reads all cache entries and sorts
> > out those with an EAPI it doesn't support. The rest gets ordered and the
> > same procedure as above applies.
> >
> > So, one of the main differences is: "reading one cache file" (if running
> > unstable you can asssume you support the highest version, thus reading
> > only one cache file) vs. "reading all cache files".
> That assumes a dumb cache format.
> Why don't we make the cache more efficient so you read one file per package /
> category / ... ?
>
> >
> > I did some performance measurements based on that. I have 1507 installed
> > packages with 5541 different versions/revisions.
> >
> > Reading from hot cache:
> > 1507 files: ~50ms
> > 5541 files: ~170ms
> >
> > Reading from cold cache:
> > 1507 files: ~2.8s
> > 5541 files: ~6s
> And now you need to pull metadata for dependency calculation. How big is the
> impact of that?
The 1507 files are the complete dep-tree cache entries for the highest
version, where the 5541 files are all the cache entries for all packages
in dep-tree.
I did say that I simplified the case a lot, didn't I? :)

>
> >
> > I made a lot of assumptions here (neglecting seek between ebuild-dir and
> > metadata-dir, other processes using the drive, 80 ebuilds from overlays
> > where the ebuild would have to be read, etc.). But estimating from the
> > numbers above I'd say that a "emerge -uD world"/"paludis -i world" will
> > be at least twice as slow, which I think is not acceptable.
> I find that quite acceptable. As long as we're using such a bad layout the
> performance is secondary.
... and I don't :)

>
> To fix the performance you'd "only" have to guarantee that the repo is
> unchanged (readonly), so you can add lots of simple caches/indexes - no need
> to source any ebuild for metadata again, one cachefile for eapi if you want
> ... I bet you find lots of small improvements that that would yield. Much more
> impressive than managing to avoid a few open() here and there ...
>
>
> > And I also don't understand your point of stating it's "bad design".
> Bad design is like smelly feet. It's hard not to notice ...
>
> > I mean: when coding you should "not optimize prematurely", but with
> > eapi-in-ebuild it is against the other principle of "not pessimize
> > prematurely" (Sutter/Alexandrescu: C++ Coding Standards).
> If you quote that try the full quote:
>
> "We should forget about small efficiencies, say about 97% of the time:
> premature optimization is the root of all evil."
>
> In other words, we should not try to make that path faster when we can avoid
> hitting it at all with a small design revision.
>
Which you still failed (after one year or so) to provide a nice cleanly
written document for.

-- 
Tiziano Müller
Gentoo Linux Developer, Council Member
Areas of responsibility:
  Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor
E-Mail   : dev-zero@g.o
GnuPG FP : F327 283A E769 2E36 18D5  4DE2 1B05 6A63 AE9C 1E30