Gentoo Archives: gentoo-dev

From: Brian Harring <ferringb@×××××.com>
To: gentoo-dev@l.g.o
Cc: ciaran.mccreesh@××××××××××.com
Subject: Re: [gentoo-dev] adding a modification timestamp to the installed pkgs database (vdb)
Date: Tue, 12 Jan 2010 23:13:18
Message-Id: 20100112231223.GC6656@hrair
In Reply to: Re: [gentoo-dev] adding a modification timestamp to the installed pkgs database (vdb) by Ciaran McCreesh
On Tue, Jan 12, 2010 at 10:12:52AM +0000, Ciaran McCreesh wrote:
> On Mon, 11 Jan 2010 15:35:51 -0700 > Denis Dupeyron <calchan@g.o> wrote: > > I'm a bit surprised by the low amount of discussions this topic has > > generated. > > There's no discussion because Brian refuses to address any comments on > the proposal and just says "we should do it anyway, and if you want it > done properly instead, do it yourself".
This is a bit of bullshit, per the norm. There is plenty of discussion- the problem is you don't like the direction it's gone. You want a whole new vdb- I don't oppose that. However I'm not interested in trying to standardize a new vdb format into PMS, at least not yet. Your argument can basically be summed up as "don't do the minimal tweak, do the whole new vdb with defined caches that all can share". Which is a fine notion, but not at all what I'm interested in- I want smething w/in the next 6 months PMs can start deploying for user benefit. I've literally seen discussions about vdb2 for >5 years now- pinning performance bets on a white elephant is not in my interest. Further, a vdb2 specified only in spce is useless w/out actual long term usage of it for performance/reliability testing. The daft thing about this is that you're ignoring one core transition issue w/ vdb2- if someone did create a vdb2, they still would need a synchronization mechanism (one quite similar to what I'm proposing). Already stated that in the bug- Repeating the relevant snippet for your benefit, consider the following chain of events- 1) portage/pkgcore support the PMS defined vdb2 while paludis doesn't 2) portage/pkgcore are invoked modifying the livefs; vdb1, vdb2 is updated. 3) paludis is invoked. vdb1 is updated, vdb2 is not 4) portage and pkgcore now cannot rely upon vdb2, since vdb1 now contains extra modifications due to paludis not supporting vdb2. To address #4, you need a way to detect that vdb1 was modified... then portage/pkgcore can synch vdb2 up w/ the changes in vdb1 (namely via basically rebuilding vdb2 from vdb1). The only real alternative to the issue above is to that of dropping vdb1- when you jump to vdb2, you drop vdb1. This means all existing tools have to know of both locations, have to support at least read on both locations- if a PM (again, paludis for the example) didn't support VDB2, under the alternative approach paludis wouldn't be usable since there would not be a vdb for it to inspect. In my opinion that is not a viable alternative- but you're free to propose whatever the hell you want. Summarizing; the synchronization primitive is needed for any future vdb2; once all vdb modifiers (paludis being the sole hold out per the norm) support updating the timestamp (and a couple of months have gone by), you get the following benefits- 1) PMs can rebuild their vdb cache validation to use that timestamp, simplifying their code and reducing redundancy in their validation 2) caches that are currently too expensive to deploy due to the cost of having to validate their way through the vdb can rely on effectively a boolean- is the timestamp newer then what they know of. This makes said caches access far faster, making them a net win (instead of a net loss)- portages vdb metadata cache comes to mind for this one for example. 3) interested parties can start developing their own vdb2, including letting users play with it while avoiding forced lock in (this is good on multiple accounts- a vdb2 existing only in a spec is worthless in performance promises compared to a vdb2 implemented and tested). Summing it up; what ciaran wants is reliant on what I'm proposing, what I'm proposing provides tangible benefits now, not years down the line as his proposal would entail. There's the lovely executive summary. ~harring


Subject Author
Re: [gentoo-dev] adding a modification timestamp to the installed pkgs database (vdb) Ciaran McCreesh <ciaran.mccreesh@××××××××××.com>