1 |
Does anybody know the point of /var/db/pkg/*/*/{[A-Z]*,*.ebuild} when |
2 |
/var/db/pkg/*/*/env*.bz2 contains all the same info minus the |
3 |
(CBUILD,CONTENTS,COUNTER,DEBUGBUILD)? |
4 |
|
5 |
DEBUGBUILD is 0 bytes most of the time. Does it really need to exist in |
6 |
production environments? |
7 |
|
8 |
These sparse files add up in size really quickly and I'd like to see |
9 |
portage shift into a direction where these things are planned out vs |
10 |
just added. I don't see any reason why portage could not simply extract |
11 |
the variables portage from the saved environment.bz2 when needed or even |
12 |
preloading them. |
13 |
|
14 |
I work on an embedded images and I have the option to dump 100% of |
15 |
/var/db/ thus having no way of knowing what's installed but I'd like to |
16 |
keep track of what's installed and not be forced into using a hacked up |
17 |
methods which post rm stuff from a target ROOT= in order to yield the |
18 |
desired result of keeping track of data without wasting valuable |
19 |
resources. |
20 |
|
21 |
Using patch2 from http://bugs.gentoo.org/show_bug.cgi?id=67190 |
22 |
And a mini subdistro meta ebuild |
23 |
http://oc12.net/~solar/gensoekris-0.0.1.ebuild |
24 |
INSTALL_MASK='*.a *.h /usr/include /usr/lib/*.o *.sample' ROOT=/tmp/root |
25 |
emerge gensoekris -K |
26 |
|
27 |
du -hs . |
28 |
12M . |
29 |
cd var/db/pkg/ |
30 |
du -hs . |
31 |
3.8M . |
32 |
# About 1/3rd of this file systems entire size is extra pkg data. |
33 |
|
34 |
# Everything else seems to be in the env so lets kill some of the temp |
35 |
fluff and do some testing. |
36 |
find . -name '[A-Z]*' -o -name '*.ebuild' | egrep -v |
37 |
'CBUILD|CONTENTS|COUNTER' | xargs rm |
38 |
|
39 |
du -hs . |
40 |
1.1M . |
41 |
|
42 |
# So it seems we could be down from 3.8 to 1.1M and still have all |
43 |
needed info for portage. |
44 |
|
45 |
# Now lets take a look at the package and how this could work. I'll use |
46 |
iptables as an exmaple. When all is said and done we want only an |
47 |
environment.bz2 left in var/db/pkg/iptables-1.2.11-r2/ |
48 |
|
49 |
ls -l $(portageq envvar PKGDIR)/All/iptables-1.2.11-r2.tbz2 |
50 |
-rw-r--r-- 1 root root 179724 Oct 10 00:53 |
51 |
/usr/portage//packages/i386-pc-linux-uclibc//All/iptables-1.2.11-r2.tbz2 |
52 |
|
53 |
# extract the portage made tarball. |
54 |
mkdir root |
55 |
tar jxf |
56 |
/usr/portage//packages/i386-pc-linux-uclibc//All/iptables-1.2.11-r2.tbz2 |
57 |
-C root/ |
58 |
du -hs root |
59 |
660K root |
60 |
|
61 |
# Repack the contents (new pkg format) |
62 |
cd root |
63 |
tar jcvf ../iptables-1.2.11-r2.tbz2 ./ |
64 |
cd ../net-firewall/iptables-1.2.11-r2/ |
65 |
bzip2 CONTENTS |
66 |
bunzip2 environment.bz2 |
67 |
echo CBUILD=$(cat CBUILD) >> environment |
68 |
echo COUNTER=$(cat COUNTER) >> environment |
69 |
bzip2 environment |
70 |
cat CONTENTS.bz2 >> environment.bz2 |
71 |
cd ../../ |
72 |
cat net-firewall/iptables-1.2.11-r2/environment.bz2 >> |
73 |
iptables-1.2.11-r2.tbz2 |
74 |
ls -l iptables-1.2.11-r2.tbz2 |
75 |
-rw-r--r-- 1 root root 174892 Nov 3 19:01 iptables-1.2.11-r2.tbz2 |
76 |
|
77 |
# As we can see the package size has become smaller. |
78 |
echo $((179724-174892)) |
79 |
4832 |
80 |
|
81 |
# Just for kix take that number and assume say about 100000 users have |
82 |
about 500 packages installed and see that across the board we could save |
83 |
the users about |
84 |
export f=$(($(($((4832*500))*100000))/1024)) |
85 |
dd if=/dev/zero of=$f count=1 bs=$f ; ls -lh $f ; rm $f |
86 |
-rw-r--r-- 1 root root r 226M Nov 3 14:19 235937500 |
87 |
# probably not really realistic as 100k of users don't have buildpkg |
88 |
enabled. |
89 |
|
90 |
# Anyway lets test the new package.tbz2. |
91 |
mkdir root |
92 |
tar jxf iptables-1.2.11-r2.tbz2 -C root ; echo $? |
93 |
0 |
94 |
du -hs root |
95 |
660K root |
96 |
# good 660K matches same as portage built tarball that was extracted |
97 |
into the root/ from before. |
98 |
|
99 |
# The du -b here could become some other tool which seeks to the second |
100 |
BZh9 header in the tarball. But to test that we could still get info out |
101 |
of the tarballs so portage could use/load it I did. |
102 |
|
103 |
tail -c $(du -b net-firewall/iptables-1.2.11-r2/environment.bz2 ) | |
104 |
bzgrep ^ARCH= |
105 |
|
106 |
# which returned. |
107 |
ARCH=x86 |
108 |
|
109 |
# now lets see how big |
110 |
rm iptables-1.2.11-r2.tbz2 |
111 |
find . -name '[A-Z]*' | xargs rm |
112 |
du -hs . |
113 |
732K . |
114 |
|
115 |
# Well sweet we just got a dir that was 3.8M down to 732K and don't |
116 |
appear to have lost anything. |
117 |
|
118 |
# New total size |
119 |
du -hs /tmp/root |
120 |
8.2M /tmp/root |
121 |
# which compresses down to about 3.6M nicely (Not bad for 36 pkgs) |
122 |
|
123 |
# another box which is my desktop is |
124 |
du -hs /var/db/pkg/ ; du -bhs /var/db/pkg/ |
125 |
83M /var/db/pkg/ |
126 |
29M /var/db/pkg/ |
127 |
# I bet most of yours are nearly as large or larger. |
128 |
|
129 |
# Cost savings here could be alot better for everybody if we moved into |
130 |
a direction which I just demonstrated. |
131 |
|
132 |
In short this all could work if we simply |
133 |
Compressed the file CONTENTS and appended it to end of environment.bz2 |
134 |
then appended environment.bz2 to the end of the ${P}.tbz2 |
135 |
|
136 |
Anybody see flaws/problems in the basic idea? |
137 |
Would this structure be suitable as the next gen pkg format? |
138 |
|
139 |
-- |
140 |
Ned Ludd <solar@g.o> |
141 |
Gentoo (hardened,security,infrastructure,embedded) Developer |