Re: [gentoo-portage-dev] Re: [RFC] Package description index file format for faster emerge search actions

[Zac Medico]

On 10/14/2014 09:27 AM, Martin Vaeth wrote:
> Zac Medico <zmedico@g.o> wrote:
>>
>> If we really want to index the homepage, then a more extensible format
>> might be better. For example, each line of the index could be a JSON
>> object like this:
>>
>> {"description": "sandbox'd LD_PRELOAD hack", "homepage":
>> "http://www.gentoo.org/proj/en/portage/sandbox/", "package_versions":
>> "sys-apps/sandbox-1.6-r2,2.3-r1,2.4,2.5,2.6-r1"}
>
> ...and when you also add some other data like LICENSE, IUSE, KEYWORDS,
> SLOT and (optionally) {P,R,}DEPEND, you essentially end up with the
> eix database (/var/cache/eix/portage.eix generated by eix-update).

After some thought, I'd prefer to stick with the simpler non-extensible
format described in my first email. Reasons include:

1) The package names and descriptions are, by far, the most commonly
searched items. So, for general use, emerge --search/--searchdesc
actions should be sufficient for most users. More advanced queries are
better suited to something like eix-db or sqlite, but the majority of
users have negligible interest in performing such advanced queries, so
it's hard to justify distributing a relatively large binary database
inside the package repository (it puts extra load on the rsync servers).
So, I think it's better to generate such databases on the client side,
using $repository/metadata/md5-cache as a source when available.

2) A plain text index, like the one I originally suggested, is small
enough (1.5 MB for current gentoo-x86) so that the additional load it
puts on the rsync servers should be manageable. Also, for repositories
distributed via a vcs such as git, changes to the plain text index will
transfer efficiently (only differences are transferred).

> The only difference is that the eix database is even somewhat more
> optimized for searching (so that e.g. the homepage and license data
> can be skipped without reading when you are looking for the description)
> and slightly compressed (common words are in a dictionary).
>
> It would make sense that portage and eix agree on a common database
> instead of caching essentially the same information twice.
> This appears especially important in view of things like eix-remote
> where this data is "merged" for various overlays without downloading
> the overlays.

I agree, but as mentioned above, I think it's better to generate such a
database on the client side, using $repository/metadata/md5-cache as a
source when available.

> You can find the description of the current format of the eix database
> in eix-db.html when you emerge eix with USE=doc.

Thanks for the info. I've reviewed the specification, and it looks like
a nice format. However, if we're going to create a shared database that
suits the needs of both portage and eix, then I would prefer to use a
general-purpose RDBMS such as sqlite (sqlightning looks interesting,
btw). Why should be go to the trouble of developing and maintaining a
special-purpose RDBMS, even though a general-purpose RDBMS suits our needs?
-- 
Thanks,
Zac