Re: [gentoo-portage-dev] Current portage well designed, but badly used

[Paul de Vrieze]

On Sunday 28 November 2004 00:10, Gustavo Barbieri wrote:
> Hello,
>
> I'm playing with portage and noticed it's well designed, but there are
> some mistakes in its usage at the moment. For example:
>
> Categories are mixed: there is a net-www/apache and net-www/mod_*
> (apache modules), but there is a more convenient category www-apache/
> for them. This is one example, there are more mistakes.  There is any
> plan to fix them in next portage releases?
>
> Some packages use numbering version padded with zero, that's good to
> list with shell functions, but it's bad because you can't change them
> to numbers and them back to string. For example:
> mail-mta/nullmailer-1.00_rc7-r4. If you Convert it to integers, it
> becomes 1.0 and you can't map back to the ebuild.
>
> Portage provides metadata.xml, cool. But it's hardly used :(
> metadata.xml seems to provide tags for maintainers, changelogs and
> long description, many (most?) packages don't use them.
>
> The portage library is too heavy, complicated and make things slow.
> Heavy and complicated I noticed from (trying to) look at the source,
> slow by usage. For example:
>
> time emerge # without parameters
> real    0m0.614s
> user    0m0.487s
> sys     0m0.046s
>
> time emerge -pv world # 16 packages to be upgraded
> real    0m22.664s
> user    0m12.423s
> sys     0m1.130s
>
> It's too much, look at debian apt, it's fast. And I can't see why
> portage is slow.
> Forgive me if I'm wrong, but portage just need to parse
> /var/lib/portage/world (237 entries in my case), them for each check
> if there is any other version greater than and if so check for
> dependencies. Why 22seconds? A hand made take less than 1.
>
>
> Also, a brief explanation on why I was playing with portage and some
> requests: I'm coding (for fun, no plan to get in a production state)
> yet another graphical package manager atop portage with the newbie in
> mind. But to achieve my goal I need:
>
>    - a fast portage. Now I'm doing a module to do this for me (see
> more above), at least the basics, like get package information,
> versions, ... and if possible resolve primary dependencies (just to
> show to user in a tab "Dependencies", hidden by default).
>
>    - more meta data, if possible a list of urls to screenshots (most
> packages have a screenshots section), if the url links to an html,
> provide a threshold of images size to get, so it connects and
> downloads every image bigger than it... cached of course.
>
>    - portage to act as a daemon, queue requests  and fetch packages.
> If portage could be a daemon with 3 threads: one that download
> packages, one that compiles and one to manage the other and accept
> requests; then it could schedule download to maximize download
> throughput, downloading smaller packages first while respecting
> dependencies, compile while download and wait until packages are there
> and the "emerge" command just send commands to it.  It would be handy
> since compiling times are huge.
>
>
> About the fast portage: I know portage is a complex monster and is the
> heart of gentoo, if it breaks, everything breaks. But how about a
> python module to be used by other packages that just want to view the
> portage and its packages. If eventually this module works as expected
> and have every current portage feature, it could replace the old one.
>    I started to code my own "fast portage", but some things are picky
> to do, and I want to know how you do that: how do you parse ebuilds to
> get USE, DESCRIPTION, SLOT, DEPEND, ... ?
>    If you want to know why my implementation is fast: I use lazy
> evaluation as far as possible. For example, I load every package, but
> the attributes to available versions, installed versions, the status,
> are just calculated on deman, I use python property() and
> setters/getters for that. Since hardly you'll use every attribute from
> everythin, it loads much faster.
>    I have preliminar code here:
> http://ltc08.ic.unicamp.br/~gustavo/packagemanager.tar.bz2, but some
> modifications I did were lost in a power outtage + xfs... I just have
> the .pyc, if someone knows how to get the .py back...

Well, as said, portage does not parse by itself, but uses bash. This is not 
really fast. The biggest issue however is the absense of lazy evaluation. 
I've been looking at it too, and even have a c++ based parser that can be 
accessed as a python module, but it's undocumented and has issues as it is 
not a full bash replacement, and ebuilds expect bash.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@g.o
Homepage: http://www.devrieze.net