Re: [gentoo-alt] tree verification in prefix

[Fabian Groffen]

Please excuse my snipping.

On 21-02-2018 21:55:29 +0100, Michael Weiser wrote:
> I've been following or rather post-reading the discussion about GLEP-74
> and it seems a lot of thought and security considerations have gone into
> it. Are you following it for your implementation?

Sure.  I have my reservations about the decisions now, but you only get
that much of detail when you start implementing such spec.

> I see you're using OpenMP. That looks very neat and could certainly make
> it scale better than gemato.

That was because I was too lazy to do pthreads stuff this time, and at
the time of writing hashgen was experimenting with OpenMP as new thing I
learnt.  The target platform rsync0 ran on (Solaris-x64 back then,
ppc64-linux now, it's been amd64-linux inbetween) always supported
OpenMP, so I could unconditionally use it.

> BTW: I remember OpenMP being plain missing from clang until recently.
> Ah, https://clang-omp.github.io/ says 3.7 and onwards have native
> support.

Yes, in the beginning it was an issue with clang on OSX, but the joy of
OpenMP in a way, it just compiles and runs way slower :)

> > hashgen currently runs in 30s or so on the tree to generate manifests.
> > I hope it can verify in the same amount of time (we're talking about a
> > Quad G5 PowerPC machine here with rusty old spinning disks), leaving it
> > in a much better position to be used for Prefix, since we tend to have
> > slower/older machines around.
>
> I can only believe that number if the portage tree fits into the fs
> cache (RAM). Then it would be purely CPU-bound, I guess, and C could
> play out its advantage.

Sure.  The ppc64 box I mention has 16GiB of ram, and 4 cores 2.5GHz
each.  14GiB of ram is used for buffers, so I may very well be a bit
over-optimistic here.

> It also shows that I have a high pain threshold in dealing with Gentoo
> on small and old machines. Considering how long Gentoo users typically
> wait for compiles to finish I wouldn't have expected a bit of a wait for
> gemato or hashverify to be much of an issue.

I have my Sun Blade 100 (sparcv9 502MHz) with 512MiB ram in mind, which
is very slow in any IO tasks, reasonable on CPU bound tasks.

> Ah, now I understand what you were getting at. I just verified: Both
> checks are working on Gentoo Linux but not in Prefix Mac. It seems to
> have to do with setting "thin-manifests" in layout.conf of the prefix
> tree. Once I set that to false, prefix portage behaves the same as
> Gentoo Linux portage.

THANKS.  I'm going to fix this immediately.

> Gentoo Linux:
> [root@linux:~] grep thin /usr/portage/metadata/layout.conf
> # Use thin Manifests for Git
> thin-manifests = false
>
> Prefix Mac:
> root@mac:/gentoo/usr/portage/xfce-base/garcon $ grep -r thin /gentoo/usr/portage/metadata/layout.conf
> # Use thin Manifests for Git
> thin-manifests = true
> root@mac:/gentoo/usr/portage/xfce-base/garcon $ sed -i -e "s,thin-manifests = true,thin-manifests = false," /gentoo/usr/portage/metadata/layout.conf

> > Now I may be all wrong with trying to implement the verification myself,
> > but that's a separate topic.  gemato should work fine.
>
> I think it's a worthy cause just for the fact that multiple
> implementations of a protocol are always good to drive improvement and
> show up the corner cases.

My thinking, plus pet-project material :)

> Maybe it shouldn't be hidden away in the prefix tree so others can more
> easily become aware of and contribute to it.

Once hashverify is done, I'll make an ebuild for it.  hashgen is not
useful in general.  Only if you create a tree yourself.

> I've retried a couple of times today. I deleted timestamp.chk several
> times and resynced until I saw no delta any more. There actually was a
> delta on the second and sometimes third run, mostly Manifest.gzs and
> metadata. It still fails reliably on second-level Manifests, e.g.
> dev-dotnet/Manifest.gz and sys-libs/Manifest.gz

Odd, was this rsync1 or rsync2?

> It looks as if regeneration of Manifests and md5-cache runs almost
> continuously, not just every 26,56 of the hour as rsync's motd says. Is
> that intentional or are perhaps multiple instances of the script running
> amok?

The script is fired off through cron at 26 and 56.  I may have gone
wrong here, but for the sake of redundancy, I scratched the
  rsync0 -> rsync{1,2}
path.  Instead rsync1 and rsync2 are doing full generation both of them.
To make this be equal, I had to play a trick with timestamps, I
basically set the timestamp to the git commit time.  Maybe this plays a
role too?

I only checked rsync1 last time, maybe rsync2 isn't as equal as I
thought.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level