Gentoo Archives: gentoo-dev

From: ross b girshick <rossgir@××××××××.edu>
To: gentoo-dev@g.o
Subject: [gentoo-dev] "Updating Portage Cache" optimizations
Date: Tue, 24 Jun 2003 15:20:04
In Reply to: Re: [gentoo-dev] Re: (FS) Attributes for Ebuilds? by Michael Kohl
1 Hi,
3 Lately I've been bothered by how long it takes to update the portage cache
4 after doing an emerge [r]sync. So I decided to dive into the portage code
5 for the first time to do something about this. What I found seems a little
6 confusing and inefficient. So I'm ask for people to clear up any
7 misconceptions I might have and get some feedback on a _simple_
8 optimization.
10 The main time siphon during the cache updating process is the function
11 portage.aux_get embedded in a double nested for loop. aux_get either
12 copies the metadata file out of /usr/portage/metadata/cache/ into
13 /var/cache/edb/dep/ or regenerates it using the ebuild if the cached
14 version is old. My laptop's hard-drive is pretty slow (4200RPM, etc) so
15 this process of copying ~ 36MB of small files takes about 4.5 minutes on
16 average. In most cases the metadata files are copied directly. I did a diff
17 on some categories in the /dep/ cache vs. the /metadata/ cache and found
18 only a few files were regenerated.
20 So my first optimization, a whopping one-liner, reduces the cache update
21 time from 4.5 minutes to 2.25 minutes on my system (and saves about 35MB
22 of disk space). Based on the code, I think a lot of other optimization can
23 be added to (such as symlinking whole category directories when there are
24 no regens in it).
26 So far I've had no problems after making this change. Can anyone think of
27 how this would introduce a bug?
29 Thanks,
30 Ross Girshick
32 p.s. I've been using gentoo for quite a while now, but I've just started
33 getting into the dev side of it. What's the proper channel for submitting
34 patches?
36 Here's the patch:
38 --- 2003-06-24 10:13:49.000000000 -0400
39 +++ 2003-06-24 10:15:04.000000000 -0400
40 @@ -3400,7 +3400,8 @@
41 if not os.path.exists(mydir):
42 os.makedirs(mydir, 2775)
43 os.chown(mydir,uid,portage_gid)
44 - shutil.copy2(mymdkey, mydbkey)
45 + #shutil.copy2(mymdkey, mydbkey)
46 + os.symlink(mymdkey, mydbkey)
47 usingmdcache=1
48 except Exception,e:
49 print "!!! Unable to copy '"+mymdkey+"' to '"+mydbkey+"'"
53 --
54 gentoo-dev@g.o mailing list


Subject Author
Re: [gentoo-dev] "Updating Portage Cache" optimizations Seemant Kulleen <seemant@g.o>