Gentoo Archives: gentoo-portage-dev

From: tvali <qtvali@×××××.com>
To: gentoo-portage-dev@l.g.o
Subject: Re: [gentoo-portage-dev] search functionality in emerge
Date: Mon, 24 Nov 2008 00:53:05
In Reply to: Re: [gentoo-portage-dev] search functionality in emerge by "René 'Necoro' Neumann"
1 There is daemon, which notices about filesystem changes -
2 would be a good choice.
4 In case many different applications use portage tree directly without using
5 any portage API (which is a bad choice, I think, and should be deprecated),
6 then there is a kind of "hack" - using
7 to create a new filesystem
8 (damn now I would like to have some time to join this game). I hope it's
9 possible to build it everywhere where gentoo should work, but it'n no
10 problem if it's not - you can implement it in such way that it's not needed.
11 I totally agree, that filesystem is a bottleneck, but this suffix trie would
12 check for directories first, I guess. Now, having this custom filesystem,
13 which actually serves portage tree like some odd API, you can have backwards
14 compability and still create your own thing.
16 Having such classes (numbers show implementation order; this is not
17 specified here if proxies are abstract classes, base classes or smth. other,
18 just it shows some relations between some imaginary objects):
20 - *1. PortageTreeApi* - Proxy for different portage trees on FS or SQL or
21 other.
22 - *2. PortageTreeCachedApi *- same, as previous, but contains boosted
23 memory cache. It should be able to save it's state, which is simply writing
24 it's inner variables into file.
25 - *3. PortageTreeDaemon *- has interface compatible with PortageTreeAPI,
26 this daemon serves portage tree to PortageTreeFS and portage tree itself. In
27 reality, it should be base class of *PortageTreeApi* and *
28 PortageTreeCachedApi* so that they could be directly used as daemons.
29 When cached API is used as daemon, it should be able to check filesystem
30 changes - thus, implementations should contain change trigger callbacks.
31 - *4. PortageTreeFS *- filesystem, which can be used to map any of those
32 to filesystem. Connectable with PortageTreeApi or PortageTreeDaemon. This
33 creates filesystems, which can be used for backwards-compability. This
34 cannot be used on architectures, which dont implement lufs-python or analog.
35 - *6. PortageTreeServer *- server, which serves data from
36 PortageTreeDaemon, PortageTreeCachedApi or PortageTreeApi to some other
37 computer.
38 - Implementations can be proxied through *PortageTreeApi*, *
39 PortageTreeCachedApi* or *PortageTreeDaemon*.
40 - *5. PortageTreeImplementationAsSqlDb*
41 - *1. PortageTreeImplementationAsFilesystem*
42 - *3. PortageTreeImplementationAsDaemon* - client, actually.
43 - *6. PortageTreeImplementationAsServer* - client, too.
45 So, *1* - creating PortageTreeApi and PortageTreeImplementationAsFilesystem
46 is pure refactoring task, at first. Then, adding more advanced functions to
47 PortageTreeApi is basically refactoring, too. PortageTreeApi should not
48 become too complex or contain any advanced tasks, which are not purely
49 db-specific, so some common baseclass could implement more high-level
50 things.
51 Then, *2* - this is finishing your schoolwork, but not yet in most powerful
52 way as we are having only index then, and first search is still slow. At
53 beginning this cache is unable to provide data about changes in portage tree
54 (which could be implemented by some versioning after this new api is only
55 place to update it), so it should have index update command and be only used
56 in search.
57 Then, *3* - having portage tree daemon means that things can really be
58 cached now and this cache can be kept in memory; also it means updates on
59 filesystem changes.
60 Then, *4* - having PortageTreeFS means that now you can easily implement
61 portage tree on faster medium without losing backwards-compability.
62 Now, *5* - implementation as SQL DB is logical as SQL is standardized and
63 common language for creating fast databases.
64 Eventually, *6* - this has really nothing to do with boosting search, but in
65 fast network it could still boost emerge by removing need for emerge --sync
66 for local networks.
68 I think that then it would be considered to have synchronization also in
69 those classes - CachedApi almost needs it to be faster with server-client
70 connections. After that, ImplementationAsSync and ImplementationAsWebRsSync
71 could be added and sync server built onto this daemon. As I really doubt
72 that emerge --sync is currently also ultraslow - I see no meaning in waiting
73 a long time to get few new items as currently seems to happen -, it would
74 boost another life-critical part of portage.
76 So, hope that helps a bit - have luck!
78 2008/11/23 René 'Necoro' Neumann <lists@××××××.eu>
81 > Hash: SHA1
82 >
83 > Mike Auty schrieb:
84 > > Finally there are overlays, and since these can change outside of an
85 > > "emerge --sync" (as indeed can the main tree), you'll have to reindex
86 > > these before each search request, or give the user stale data until they
87 > > manually reindex.
88 >
89 > Determining whether there has been a change to the ebuild system is a
90 > major point in the whole thing. What does a great index serves you, if
91 > it does not notice the changes the user made in his own local overlay?
92 > :) Manually re-indexing is not a good choice I think...
93 >
94 > If somebody comes up here with a good (and fast) solution, this would be
95 > a nice thing ;) (need it myself).
96 >
97 > Regards,
98 > René
99 > -----BEGIN PGP SIGNATURE-----
100 > Version: GnuPG v2.0.9 (GNU/Linux)
101 > Comment: Using GnuPG with Mozilla -
102 >
104 > 3CoAn2wV6g8/8uj+T99cxJGdQBxTtZjI
105 > =2I2j
106 > -----END PGP SIGNATURE-----
107 >
108 >
111 --
112 tvali
114 Kuskilt foorumist: - kui inglise keelt oskad.
115 Muide, üle 120 oled väga tark, üle 140 oled geenius, mingi 170 oled ju mingi
116 täica pea nagu prügikast...


Subject Author
Re: [gentoo-portage-dev] search functionality in emerge "René 'Necoro' Neumann" <lists@××××××.eu>