Re: [gentoo-dev] adding a modification timestamp to the installed pkgs database (vdb)

[Brian Harring]

On Tue, Jan 12, 2010 at 10:12:52AM +0000, Ciaran McCreesh wrote:
> On Mon, 11 Jan 2010 15:35:51 -0700
> Denis Dupeyron <calchan@g.o> wrote:
> > I'm a bit surprised by the low amount of discussions this topic has
> > generated.
>
> There's no discussion because Brian refuses to address any comments on
> the proposal and just says "we should do it anyway, and if you want it
> done properly instead, do it yourself".

This is a bit of bullshit, per the norm.  There is plenty of 
discussion- the problem is you don't like the direction it's gone.  
You want a whole new vdb- I don't oppose that.  However I'm not 
interested in trying to standardize a new vdb format into PMS, at 
least not yet.

Your argument can basically be summed up as "don't do the minimal 
tweak, do the whole new vdb with defined caches that all can share".   

Which is a fine notion, but not at all what I'm interested in- I want 
smething w/in the next 6 months PMs can start deploying for user 
benefit.  I've literally seen discussions about vdb2 for >5 years now- 
pinning performance bets on a white elephant is not in my interest.  
Further, a vdb2 specified only in spce is useless w/out actual long 
term usage of it for performance/reliability testing.


The daft thing about this is that you're ignoring one core transition 
issue w/ vdb2- if someone did create a vdb2, they still would need a 
synchronization mechanism (one quite similar to what I'm proposing).  
Already stated that in the bug-

http://bugs.gentoo.org/show_bug.cgi?id=290428#c11

Repeating the relevant snippet for your benefit, consider the 
following chain of events-

1) portage/pkgcore support the PMS defined vdb2 while paludis doesn't
2) portage/pkgcore are invoked modifying the livefs; vdb1, vdb2 is 
updated.
3) paludis is invoked.  vdb1 is updated, vdb2 is not
4) portage and pkgcore now cannot rely upon vdb2, since vdb1 now 
contains extra modifications due to paludis not supporting vdb2.


To address #4, you need a way to detect that vdb1 was modified... then 
portage/pkgcore can synch vdb2 up w/ the changes in vdb1 (namely via 
basically rebuilding vdb2 from vdb1).

The only real alternative to the issue above is to that of dropping 
vdb1- when you jump to vdb2, you drop vdb1.  This means all existing 
tools have to know of both locations, have to support at least read on 
both locations- if a PM (again, paludis for the example) didn't 
support VDB2, under the alternative approach paludis wouldn't be 
usable since there would not be a vdb for it to inspect.

In my opinion that is not a viable alternative- but you're free to 
propose whatever the hell you want.


Summarizing; the synchronization primitive is needed for any future 
vdb2; once all vdb modifiers (paludis being the sole hold out per the 
norm) support updating the timestamp (and a couple of months have 
gone by), you get the following benefits-

1) PMs can rebuild their vdb cache validation to use that timestamp, 
simplifying their code and reducing redundancy in their validation
2) caches that are currently too expensive to deploy due to the cost 
of having to validate their way through the vdb can rely on 
effectively a boolean- is the timestamp newer then what they know of.  
This makes said caches access far faster, making them a net win 
(instead of a net loss)- portages vdb metadata cache comes to mind for 
this one for example.
3) interested parties can start developing their own vdb2, including 
letting users play with it while avoiding forced lock in (this is good 
on multiple accounts- a vdb2 existing only in a spec is worthless in 
performance promises compared to a vdb2 implemented and tested).


Summing it up; what ciaran wants is reliant on what I'm proposing, 
what I'm proposing provides tangible benefits now, not years down the 
line as his proposal would entail.

There's the lovely executive summary.
~harring