| 1 |
On 30-06-2020 13:13:29 -0500, Sid Spry wrote: |
| 2 |
> On Tue, Jun 30, 2020, at 1:20 AM, Fabian Groffen wrote: |
| 3 |
> > Hi, |
| 4 |
> > |
| 5 |
> > On 29-06-2020 21:13:43 -0500, Sid Spry wrote: |
| 6 |
> > > Hello, |
| 7 |
> > > |
| 8 |
> > > I have some runnable pseudocode outlining a faster tree verification algorithm. |
| 9 |
> > > Before I create patches I'd like to see if there is any guidance on making the |
| 10 |
> > > changes as unobtrusive as possible. If the radical change in algorithm is |
| 11 |
> > > acceptable I can work on adding the changes. |
| 12 |
> > > |
| 13 |
> > > Instead of composing any kind of structured data out of the portage tree my |
| 14 |
> > > algorithm just lists all files and then optionally batches them out to threads. |
| 15 |
> > > There is a noticeable speedup by eliding the tree traversal operations which |
| 16 |
> > > can be seen when running the algorithm with a single thread and comparing it to |
| 17 |
> > > the current algorithm in gemato (which should still be discussed here?). |
| 18 |
> > |
| 19 |
> > I remember something that gemato used to use multiple threads, but |
| 20 |
> > because it totally saturated disk-IO, it was brought back to a single |
| 21 |
> > thread. People were complaining about unusable systems. |
| 22 |
> > |
| 23 |
> |
| 24 |
> I think this is an argument for cgroups limits support on the portage process or |
| 25 |
> account as opposed to an argument against picking a better algorithm. That is |
| 26 |
> something I have been working towards, but I am only one man. |
| 27 |
|
| 28 |
But this requires a) cgroups support, and b) the privileges to use it. |
| 29 |
Shouldn't be a problem in the normal case, but just saying. |
| 30 |
|
| 31 |
> > In any case, can you share your performance results? What speedup did |
| 32 |
> > you see, on warm and hot FS caches? Which type of disk do you use? |
| 33 |
> > |
| 34 |
> |
| 35 |
> I ran all tests multiple times to make them warm off of a Samsung SSD, but |
| 36 |
> nothing very precise yet. |
| 37 |
> |
| 38 |
> % gemato verify --openpgp-key signkey.asc /var/db/repos/gentoo |
| 39 |
> [...] |
| 40 |
> INFO:root:Verifying /var/db/repos/gentoo... |
| 41 |
> INFO:root:/var/db/repos/gentoo verified in 16.45 seconds |
| 42 |
> |
| 43 |
> sometimes going higher, closer to 18s, vs. |
| 44 |
> |
| 45 |
> % ./veriftree.py |
| 46 |
> 4.763171965983929 |
| 47 |
> |
| 48 |
> So roughly an order of magnitude speedup without batching to threads. |
| 49 |
|
| 50 |
That is kind of a change. Makes one wonder if you really did the same |
| 51 |
work. |
| 52 |
|
| 53 |
> > You could compare against qmanifest, which uses OpenMP-based |
| 54 |
> > paralllelism while verifying the tree. On SSDs this does help. |
| 55 |
> > |
| 56 |
> |
| 57 |
> I lost my notes -- how do I specify to either gemato or qmanifest the GnuPG |
| 58 |
> directory? My code is partially structured as it is because I had problems doing |
| 59 |
> this. I rediscovered -K/--openpgp-key in gemato but am unsure for qmanifest. |
| 60 |
|
| 61 |
qmanifest doesn't do much magic out of the standard gnupg practices. |
| 62 |
(It is using gpgme.) If you want it to use a different gnupg dir, you |
| 63 |
may change HOME, or GNUPGHOME. |
| 64 |
|
| 65 |
Thanks, |
| 66 |
Fabian |
| 67 |
|
| 68 |
-- |
| 69 |
Fabian Groffen |
| 70 |
Gentoo on a different level |
Archives