Gentoo Archives: gentoo-portage-dev

From: Bastian Balthazar Bux <BastianBalthazarBux@×××××××××.it>
To: gentoo-portage-dev@l.g.o
Subject: Re: [gentoo-portage-dev] Cache rewrite backport
Date: Wed, 12 Oct 2005 12:57:43
Message-Id: 434D0806.50307@pnpitalia.it
In Reply to: Re: [gentoo-portage-dev] Cache rewrite backport by Brian Harring
1 Brian Harring ha scritto:
2 > Rather then keep posting large patches here, just going to post them
3 > to dev.gentoo.org/~ferringb/portage/2.0 ... for example,
4 > http://dev.gentoo.org/~ferringb/portage/2.0/3.0-cache-backport-experimental-4.patch
5 >
6 > should be pulled rather then previous patch.
7 >
8 > Helluva lot easier for me since I can correct dumb ass mistakes like
9 > an import pdb for testing sliding into a patch :)
10 >
11 > Think that's probably a sign it's time for some sleep...
12 > ~harring
13
14 premessa:
15 emerge metadata does not recreate the cache after a
16 "rm -rf $PORTDIR/metadata/cache/*" .
17 The first thing that came in mind is that I'm missing something obvious.
18
19 Today working on a different box, slower cpu, faster disk (ratio 2:1)
20
21 sys-apps/portage-2.0.53_rc5 patched version
22 time emerge --metadata three times, first time with error trace
23 real 3m43.162s 0m14.986s 0m14.124s
24 user 0m13.580s 0m10.790s 0m11.150s
25 sys 0m7.990s 0m3.280s 0m2.960s
26
27 sys-apps/portage-2.0.53_rc5 vanilla
28 time emerge --metadata three times
29 real 0m47.913s 0m52.677s 0m37.448s
30 user 0m32.900s 0m31.720s 0m31.870s
31 sys 0m7.020s 0m6.440s 0m5.530s
32
33 sys-apps/portage-2.0.53_rc5 patched version again
34 time emerge --metadata three times, first time with error trace
35 real 0m17.386s 0m14.029s 0m13.873s
36 user 0m12.110s 0m10.980s 0m10.820s
37 sys 0m3.950s 0m3.020s 0m3.040s
38
39 =================== strace patched on last run
40 grep -c "..." strace
41 patched vanilla
42 ^mmap2 11642 11218
43 ^open 14056 35048
44 ^read 3440 44880
45 ^fstat64 14269 76496
46 ^stat64 127923 180898
47
48 =================== cache file example
49 cat $PORTDIR/metadata/cache/sys-apps/portage-2.0.53_rc5
50 >=dev-lang/python-2.2.1
51 !build? ( >=sys-apps/sed-4.0.5 dev-python/python-fchksum
52 >=dev-lang/python-2.2.1 userland_GNU? ( sys-apps/debianutils )
53 >=app-shells/bash-2.05a ) !userland_Darwin? ( sys-apps/sandbox )
54 selinux? ( >=dev-python/python-selinux-2.15 )
55 0
56 mirror://gentoo/portage-2.0.53_rc5.tar.bz2
57 http://dev.gentoo.org/~jstubbs/releases/portage-2.0.53_rc5.tar.bz2
58
59 http://www.gentoo.org/
60 GPL-2
61 The Portage Package Management System. The primary package management
62 and distribution system for Gentoo.
63 ~alpha ~amd64 ~arm ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc-macos ~ppc64 ~s390
64 ~sh ~sparc ~x86
65 multilib toolchain-funcs
66 build selinux
67 =================== cache file wc
68 wc $PORTDIR/metadata/cache/sys-apps/portage-2.0.53_rc5
69 22 57 641 /usr/portage/metadata/cache/sys-apps/portage-2.0.53_rc5
70
71 =================== 1st run backtrace
72 >>> Updating Portage cache: 100%
73 File "/usr/bin/emerge", line 2740, in ?
74 mybestpv=portage.portdb.xmatch("bestmatch-visible","sys-apps/portage")
75 File "/usr/lib/portage/pym/portage.py", line 5486, in xmatch
76 myval=best(self.xmatch("match-visible",None,mydep=mydep,mykey=mykey))
77 File "/usr/lib/portage/pym/portage.py", line 5497, in xmatch
78
79 myval=match_from_list(mydep,self.xmatch("list-visible",None,mydep=mydep,mykey=mykey))
80 File "/usr/lib/portage/pym/portage.py", line 5483, in xmatch
81 myval=self.gvisible(self.visible(self.cp_list(mykey)))
82 File "/usr/lib/portage/pym/portage.py", line 5581, in gvisible
83 keys, eapi = db["/"]["porttree"].dbapi.aux_get(mycpv, ["KEYWORDS",
84 "EAPI"])
85 File "/usr/lib/portage/pym/portage.py", line 5215, in aux_get
86 elif len(mydata.get("_eclasses_", [])) > 0:
87 File "/usr/lib/python2.4/UserDict.py", line 158, in get
88 return self[key]
89 File "/usr/lib/portage/pym/cache/mappings.py", line 78, in __getitem__
90 self.d.update(self.pull())
91 File "/usr/lib/portage/pym/cache/flat_hash.py", line 29, in callit
92 return args[0](*args[1:]+args2)
93 File "/usr/lib/portage/pym/cache/flat_hash.py", line 47, in _pull
94 raise cache_errors.CacheCorruption(cpv, e)
95 File "/usr/lib/portage/pym/cache/cache_errors.py", line 18, in __init__
96 import traceback;traceback.print_stack()
97 File "/usr/bin/emerge", line 2740, in ?
98 mybestpv=portage.portdb.xmatch("bestmatch-visible","sys-apps/portage")
99 File "/usr/lib/portage/pym/portage.py", line 5486, in xmatch
100 myval=best(self.xmatch("match-visible",None,mydep=mydep,mykey=mykey))
101 File "/usr/lib/portage/pym/portage.py", line 5497, in xmatch
102
103 myval=match_from_list(mydep,self.xmatch("list-visible",None,mydep=mydep,mykey=mykey))
104 File "/usr/lib/portage/pym/portage.py", line 5483, in xmatch
105 myval=self.gvisible(self.visible(self.cp_list(mykey)))
106 File "/usr/lib/portage/pym/portage.py", line 5581, in gvisible
107 keys, eapi = db["/"]["porttree"].dbapi.aux_get(mycpv, ["KEYWORDS",
108 "EAPI"])
109 File "/usr/lib/portage/pym/portage.py", line 5215, in aux_get
110 elif len(mydata.get("_eclasses_", [])) > 0:
111 File "/usr/lib/python2.4/UserDict.py", line 158, in get
112 return self[key]
113 File "/usr/lib/portage/pym/cache/mappings.py", line 78, in __getitem__
114 self.d.update(self.pull())
115 File "/usr/lib/portage/pym/cache/flat_hash.py", line 29, in callit
116 return args[0](*args[1:]+args2)
117 File "/usr/lib/portage/pym/cache/flat_hash.py", line 47, in _pull
118 raise cache_errors.CacheCorruption(cpv, e)
119 File "/usr/lib/portage/pym/cache/cache_errors.py", line 18, in __init__
120 import traceback;traceback.print_stack()
121
122 =================== trace of normal emerge
123 # emerge -av portage
124
125 These are the packages that I would merge, in order:
126
127 Calculating dependencies File "/usr/bin/emerge", line 3111, in ?
128 retval,favorites=mydepgraph.select_files(myfiles)
129 File "/usr/bin/emerge", line 1117, in select_files
130 self.mysd = self.select_dep(portage.root,mykey,arg=x)
131 File "/usr/bin/emerge", line 1226, in select_dep
132 myeb_matches = portage.portdb.xmatch("match-visible",x)
133 File "/usr/lib/portage/pym/portage.py", line 5497, in xmatch
134
135 myval=match_from_list(mydep,self.xmatch("list-visible",None,mydep=mydep,mykey=mykey))
136 File "/usr/lib/portage/pym/portage.py", line 5483, in xmatch
137 myval=self.gvisible(self.visible(self.cp_list(mykey)))
138 File "/usr/lib/portage/pym/portage.py", line 5581, in gvisible
139 keys, eapi = db["/"]["porttree"].dbapi.aux_get(mycpv, ["KEYWORDS",
140 "EAPI"])
141 File "/usr/lib/portage/pym/portage.py", line 5215, in aux_get
142 elif len(mydata.get("_eclasses_", [])) > 0:
143 File "/usr/lib/python2.4/UserDict.py", line 158, in get
144 return self[key]
145 File "/usr/lib/portage/pym/cache/mappings.py", line 78, in __getitem__
146 self.d.update(self.pull())
147 File "/usr/lib/portage/pym/cache/flat_hash.py", line 29, in callit
148 return args[0](*args[1:]+args2)
149 File "/usr/lib/portage/pym/cache/flat_hash.py", line 47, in _pull
150 raise cache_errors.CacheCorruption(cpv, e)
151 File "/usr/lib/portage/pym/cache/cache_errors.py", line 18, in __init__
152 import traceback;traceback.print_stack()
153 ...done!
154 [ebuild R ] sys-apps/portage-2.0.53_rc5 -build (-selinux) 0 kB
155 =================== hw/sw info
156 hdparm -tT /dev/hda3
157 Timing cached reads: 676 MB in 2.00 seconds = 338.00 MB/sec
158 Timing buffered disk reads: 122 MB in 3.04 seconds = 40.13 MB/sec
159
160 dev-lang/python: 2.3.5, 2.4.2
161 sys-apps/sandbox: 1.2.13
162
163 cat /proc/cpuinfo
164 processor : 0
165 vendor_id : AuthenticAMD
166 cpu family : 6
167 model : 7
168 model name : AMD Duron(tm) Processor
169 stepping : 1
170 cpu MHz : 1211.995
171 cache size : 64 KB
172 fdiv_bug : no
173 hlt_bug : no
174 f00f_bug : no
175 coma_bug : no
176 fpu : yes
177 fpu_exception : yes
178 cpuid level : 1
179 wp : yes
180 flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca
181 cmov pat pse36 mmx fxsr sse syscall mmxext 3dnowext 3dnow
182 bogomips : 2427.41
183
184 --
185 gentoo-portage-dev@g.o mailing list

Replies

Subject Author
Re: [gentoo-portage-dev] Cache rewrite backport Brian Harring <ferringb@g.o>