Gentoo Archives: gentoo-portage-dev

From: Zac Medico <zmedico@g.o>
To: gentoo-portage-dev@l.g.o
Subject: Re: [gentoo-portage-dev] Google SoC and "cache sync"
Date: Fri, 03 Apr 2009 05:57:23
In Reply to: Re: [gentoo-portage-dev] Google SoC and "cache sync" by Emma Strubell
2 Hash: SHA1
4 Emma Strubell wrote:
5 > Actually, do you think more work on faster search would be an adequate
6 > project for soc?
8 I don't think it would be enough work for a soc project.
10 > In order for it to actually be a plausible
11 > modification to portage I still need to implement regex search and
12 > support for overlays. As I realized too soon before I had to submit
13 > the project to my prof, integrating regex search into the
14 > suffix-tree-like index that I created would require a pretty
15 > substantial overhaul of my implementation, if not a separate
16 > implementation to deal with regex queries. I'm sure it would be
17 > possible though. The biggest problem remaining with my implementation,
18 > as I believe I said before, is that cPickle unpickles (and pickles?)
19 > the index wayy to slowly. Without a better serialization module my
20 > implementation is pretty much useless, but ignoring the time it takes
21 > to unpickle the index, my implementation is something like an order of
22 > magnitude faster than the current search implementation. That's
23 > promising, right?
25 Well, your serialization problem is somewhat surprising. How big is
26 that index file? Considering the speed with which esearch is able to
27 load and search its 2 MB database, I suspect that the approach
28 you're using may not be optimal for the data set.
30 > I haven't tried out any other picklers yet so I'm
31 > not sure how much of an improvement it might be possible to get. I
32 > sure, however, that writing my own superior pickler would be beyond my
33 > abilities. Unless serialization is simpler than I think it is, and I
34 > could maybe throw together something that is optimized for this
35 > specific data structure.
36 - --
37 Thanks,
38 Zac
40 Version: GnuPG v2.0.11 (GNU/Linux)
43 /GoAnAgZc9zuodpL4FFj4mvMZ5JdpEwk
44 =+aJx
45 -----END PGP SIGNATURE-----