Re: [gentoo-dev] Re: Bug #565566: Why is it still not fixed?

[Rich Freeman]

On Fri, Feb 26, 2016 at 7:59 AM, Martin Vaeth <martin@×××××.de> wrote:
> Rich Freeman <rich0@g.o> wrote:
>>>
>>> And currently the git history is still almost empty...
>>>
>>
>> If you want pre-migration history you need to fetch that separately.
>
> How? Neither on gitweb.gentoo.org nor on github I found an obvious
> repository with this data.

https://wiki.gentoo.org/wiki/Gentoo_git_workflow#Grafting_Gentoo_History_Onto_the_Active_Repo

If you're interested in history it is easy to do, and the repo on
github works fine for web access or the various github stats/etc.
Well, sort-of - I get the impression that github doesn't host a lot of
repos with that much history and when you push that repo to github for
the first time it will timeout and die and the repo will appear on the
site 30-60min later (I imagine subsequent pushes would be fine).  I
think we actually have one of the largest git repos out there in terms
of number of objects.  At least, when I was keeping tabs on other
migration efforts there weren't many that came close (including some
projects that you'd think of as having a lot of history).  The fact
that every package revision+patch+etc is a file in Gentoo is a big
part of that.

>
>> It is about 1.7G.
>> Considering that this represents a LOT more than 2-3 years of history
>
> If the 1.7G are fully compressed history, this would confirm
> my estimate rather precisely, if it represents (1700/120 - 1) ~ 13 years.

Perhaps I misread your post then.  I saw lots of numbers but not many
units, and I probably didn't follow what you intended to say.

>
> Note that I compared squashfs with a git user who does not even
> care about git-internal recompression. Of course, you can decrease
> the factor somewhat if e.g. your checked-out tree is still stored
> on squashfs. This does not change the fact that the factor will
> increase every year by about 1 (or probably more, because git
> uses the uneffective gzip compression, only).
>

A checkout of gentoo-x86 is about 590M.  If you use the repo that
includes cache/etc it expands to 1.2G.  13 years of history is 1.7G.
Clearly it doesn't increase by a factor of 1 every year, unless again
I'm misunderstanding what you're intending to communicate.

A git checkout consists of two parts.  It has the .git directory which
contains all the data, and it consists of the working tree.  In the
case of gentoo-x86 the working tree is about 440MB and the history is
about 150M.

The working tree doesn't really change in size much - it just reflects
the size of the current revision of the tree. It is also not
compressed (unless you stick the whole thing in a squashfs, which you
could certainly do).  It is the history which continuously grows.
However, the history IS compressed and the reality is that most new
ebuilds are similar to ebuilds that are already in the history, so it
compresses very well.  Of course it would be nice if you could use
something other than gzip to compress it.

There is no reason that somebody couldn't distribute squashfs versions
of a git /usr/portage, but if you want the full history it would still
be around 1.7G.  It would still be smaller than a checked-out tree
(the 1.7G figure is just history - it doesn't include the extra 440MB
or so for the checkout).

My point wasn't so much that there aren't sized benefits to squashfs
and no history.  I'm just saying that git is pretty efficient for what
it does do.

-- 
Rich