[gentoo-dev] Gentoo XML Database: More Data

[Yannick Koehler]

On February 7, 2003 12:34 pm, Yannick Koehler wrote:
> Discussion with carspaski reveal thought that the use of the database will
> actually not speed up emerge.  Because emerge loads the cache inside an
> internal memory database and python allow him to leave that in memory in
> between runs making it very fast and efficient as only the require entry of
> the database gets loaded instead of the whole database.

Just a note about that comment I made.  When I said that it wouldn't speed-up 
emerge it was related to specific functions.  For example, if you do for the 
first time:

emerge kde

Emerge will then fetch the kde ebuild get the dependency and fetch all the 
dependency.  Cache this information inside its internal persistent db and 
then execute the operation.

In a DB mode, the database need to be loaded in some way or the index.  It is 
hard to imagine that the number of I/O will actually be less than the current 
one described above.

But, there is cases where a db would speed up portage and that's why the xml 
file is getting interesting.  It allow to import into a db who knows about 
xml or try things using xml/text related tools.

> Some benefit I see from the xml db is for side-tools, for example search
> description of ebuilds is faster when using xml db as it is a single file
> and software only look for string that start with <description>.  One can
> use grep/regexp to do such query or built an xml capable application.

I have done the following experiment.  When I posted the original mail, I 
generated the gentoo.xml file.  The file was actually 4762563 bytes.  I did 
report it was 9525071 bytes but this was wrong.  My script had generated a 
double file...  I found this by using

grep "version name=\"kdelibs-3.1" gentoo.xml

which outputted two instance.  After correcting the file was smaller.   
Something that got my attention also is that generating a gentoo.xml today 
got me a 4852253 bytes file.  Now if you compare the date/size:

2003-02-07 12:34 -> 4762563 bytes
2003-02-11 09:10 -> 4852253 bytes

This is a 89680 bytes difference.  Running emerge rsync daily is giving me 
more than 1.2 megs a shot.  So quick calculation:

11 Feb - 7 Feb = 4 days.
4 * 1.2 megs = 4.8 megs
4 * 89k = 356k
4.8 - 356k = ~4.45 megs

Which means that, if the gentoo.xml contained all the info required for 
calculating the dependencies and only fetching required ebuilds which I'm 
pretty sure it does, would mean that I have wasted 4.45 megs of bandwidth 
this past 4 days.

Now consider that I'm not alone and that in my case I have a shared repository 
both at home and at work, this is a huge waste of bandwidth for only 4 
days...  And those are bytes, not bits...

Other information...

ykoehler@corneille ykoehler $ time grep "version name=" -c gentoo.xml 
gentoo2.xml
gentoo.xml:7262
gentoo2.xml:7373

real    0m0.107s
user    0m0.010s
sys     0m0.050s

It takes less than 1 seconds for grep to parse the file and retrieve all 
occurrences of version name=".  While it is true that grep doesn't generate 
data structure in memory and parse the inner part of the version tag, this 
actually make me re-think my original discussion with carpaski where we got 
to the conclusion that the speedup for emerge would be minimal.  I now 
actually think that given a proper xml file which would minimize even more 
parsing by adding to the xml file generation has actually possibility for 
huge saving in bandwidth, hard disk space and speed on a local pc.

It also take less time to generate the xml file than to issue an emerge rsync 
for a 1 day of changes.  For example, I have run emerge rsync this morning:

rsync[15612] (receiver) heap statistics:
  arena:        5362104   (bytes from sbrk)
  ordblks:          551   (chunks not in use)
  smblks:             2
  hblks:              1   (chunks from mmap)
  hblkhd:        258048   (bytes from mmap)
  usmblks:            0
  fsmblks:           40
  uordblks:     4667912   (bytes used)
  fordblks:      694192   (bytes free)
  keepcost:       53464   (bytes in releasable chunk)

Number of files: 36464
Number of files transferred: 14444
Total file size: 29101431 bytes
Total transferred file size: 14445832 bytes
Literal data: 69 bytes
Matched data: 14445763 bytes
File list size: 848703
Total bytes written: 408076
Total bytes read: 1349498

wrote 408076 bytes  read 1349498 bytes  1858.88 bytes/sec
total size is 29101431  speedup is 16.56

>>> Updating Portage cache...


real    16m19.682s
user    0m21.990s
sys     0m40.790s

So 16 min. later and 1.3 megs I got the database update for today...

corneille dep # time /home/ykoehler/test.sh >/home/ykoehler/gentoo3.xml

real    1m17.284s
user    0m21.780s
sys     0m32.350s

Generated the gentoo.xml file at size 4865557 bytes.  Which if I had rsynced 
from the server would have transform into (4865557 - 4852253) ~13304 bytes.

I heard that there was already works on getting portage to use a real db such 
as berkeley or mysql etc..  In any case I think the distribution format that 
make most sense is xml.  This format can easily be manipulated using xslt to 
fit the need of many people and can be use with mostly any existing text 
tools.  Using xsl it also can be nicely converted into HTML and a dependency 
tree is easily built from there.  RSync could quickly figure out which part 
changes and update those in a fraction of time it takes today to diff the 
trees.

Hard Info:
P3 800 Mhz
Slow IDE Hard Disk 5400 rpm
256 megs ram
i810 chipset

So, with those numbers now extracted, I'm about to attempt to move emerge to 
this system on my PC and get you more "real" numbers using that mode.

-- 

Yannick Koehler

--
gentoo-dev@g.o mailing list