Re: [gentoo-dev] A few modest suggestions regarding tree size

[Ed Grimm]

On Thu, 14 Oct 2004, Luke-Jr wrote:
> On Thursday 14 October 2004 4:35 pm, Roman Gaufman wrote:
>> On Thu, 14 Oct 2004 16:30:29 +0000, Luke-Jr <luke-jr@×××××××.org> wrote:
>>> On Thursday 14 October 2004 2:49 pm, Ciaran McCreesh wrote:
>>>> On Thu, 14 Oct 2004 07:43:11 -0700 Mark Dierolf <mark@×××.com> wrote:
>>>>| I've been watching this discussion as far as tree size, and i'm
>>>>| suprised nobody has brought the idea of on-demand downloading yet.

You should've watched closer.  I did not mention true on-demand
downloading, because of having seen in the archive the last time it was
discussed and dismissed.  I think that what I proposed would probably be
quicker than downloading the metadata, it doesn't deviate from the
concepts that have already made it into code so widely, and it hasn't
been rejected four times yet.

To reitterate, my idea was that you're probably most interested in the
packages you've already installed; so have an option to just sync
particular files, to complement the option of not syncing particular
files.

>> Huh? -- name 1 binary distribution that does that? -- all of the ones
>> I tried fetch a list of available packages -- which is exactly what
>> the portage tree provides.
>
> Why would they need a list of available packages? Such a list is
> useful *only* to the user. apt-get, ipkg, and urpmi are going to know
> the package name beforehand.

How do these programs accomplish that?  They request a list of available
packages.

>>> On Thursday 14 October 2004 3:14 pm, Patrick Lauer wrote:
>>>> So you only have to rsync the dependency info. You save maybe 50%
>>>> traffic, but need some ebuild servers that will be hit by millions
>>>> of small requests for single ebuilds. No thanks.
>>>
>>> Actually, you don't even need to sync that. Simply download the
>>> primary ebuild, read the dep info, download the next one, etc. Most
>>> modern versions of file transfer protocols (HTTP and FTP, at least;
>>> don't know about rsync) support multiple transfers in a single
>>> connection.
>>
>> How would it know what ebuild to fetch exactly?  --- just think about
>> that for a second.

The metadata files list dependancies, keywords, a description.  It would
be technically feasible to do the dependancy evaluation and ebuild
selection for the entire ebuild session just using metadata, and have a
single medium rsync connection per emerge run.  However, I couldn't code
it in Python, and I can't really explain it in English.

> ebuild doesn't deal with dependencys anyway, AFAIK. emerge would need
> the fetching functionality and could figure out the name based on
> (originally) the user's specification and (for deps) the DEPEND
> contents themselves.  Portage *already* needs to know what the name of
> the package is anyway.

ebuild files are the ultimate source of the dependancy information.  The
point on your side is that they're not the sole repository of same;
someone saw fit to export that data into cache files, so one could use
those cache files for your goal.

> On Thursday 14 October 2004 4:41 pm, Georgi Georgiev wrote:
>> the part where the http and ftp internals get handled by portage
>> internally, instead of handling them to an external program like
>> wget, are the reason why the idea was dismissed as unworkable several
>> times before.
>
> Not really a good excuse. HTTP isn't an overly complicated protocol.
> Including the fetching functionality also has other advantages, such
> as one less program to depend on (and thus one fewer that can be
> broken and screw up Portage).

I think the part that probably intimidates them is where we're
processing a particular list of stuff, and then we decide we want to get
more stuff.  This basically requires explicit threading to pull it off
properly; it also requires a mindset that can deal with threading.  As
someone with such a mindset, I can confidently say, no one writes that
kind of code without good cause.  As an example, email servers could
definitely use this type of code, but most of them, including sendmail,
do not use it.


Luke, do you have the coding ability to write the changes that would be
required to get something like this to work?  I ask, because I think
what would be needed for you to convince anyone would be a proof of
concept, which made at most one connection to a mirror.  Until you have
such a thing, the prior ideas that have been discussed (which, despite
my having found the previous discussion, I did not find, as that was
another, "this has been discussed before" dismissal) are much firmer in
their minds than anything you are presenting, and I don't think you're
going to overcome that.

In any event, I think that you and I, and anyone else interested in
having this happen should get together off the list, outside of gentoo
discussion space.  The idea is only partially formed, and none of the
devs are going to be convinced by anything less than a full plan that
addresses all of their concerns, although I think a working prototype
would be better.  (You may think your idea is complete, but it could not
be coded simply on the ideas that have been discussed on this list over
the past couple of weeks.  What we need is something thorough enough to
both build the code and demonstrate to all that it won't make the
infrastructure hurt.  By the way, the only way to do that is to prove
that it will actually reduce infrastructure load.)

Ed

--
gentoo-dev@g.o mailing list