[gentoo-lisp] [RFC] A couple of ideas

[Tomás Touceda]

Hello everyone,

So a couple of days ago we discussed with pchrist about how we will
handle porting almost everything in the lisp-overlay to the main tree.
We need feedback on the following ideas to solve this, so please critic
and comment on this.

The first idea is to move all the cl-* packages from the main tree to a
new overlay different from the official one, so we can have a clean
start in the main tree. Since most of these packages are broken (meaning
bad src_uri, bad deps, and who knows what else at this point), we
thought that this won't hurt the users. Most of us are using what's in
the overlay (and whoever isn't, please do), but either way the packages
aren't going to disappear they're just being moved, so don't panic :).

The second idea is for mirroring all the tarballs that the cl-* packages
use. Why do this? Well, most cl packages don't have a versioned tarball,
so this brings a couple of problems regarding digests, and basic version
control, so what we want to do is to build a little system to facilitate
this job.

Here's what I've written about this second idea:

****** Start ******

The problem:

Lisp packages' upstream don't name their tarballs with version numbers.
When there's a new release, the tarball digest changes, and then the
package digest fails.

A posible solution:

Build an app that controls when the tarballs are updated and notifies
us about it in some way, and that provides a way of mirroring the tarball.

Application design:

The idea is to have a record of the tarball url, md5 digest, and the
package name.
The app should provide a way to:
    (Absolutely necessary)
    1) Add a new tarball providing only the url and the name.
    2) Delete a tarball selecting an index from a list.
    3) Download every tarball, compute its md5 digest and notify if it
    doesn't match with what's stored.
    4) Given the new version, mirror the tarball with the form ${PN}-${PV}.

    (Possible)
    5) A way to check what's the new version (I don't see this possible in
    a generic way).
    6) A way to backup the package information (if we use an XML file, this
    should be a must)
    7) A way to clean old tarballs, or the ones that are already mirrored
    in Gentoo.

An easy way to store the tarball's data would be in an XML. A possible
DTD could be:

<?xml version="1.0"?>
<!DOCTYPE packages [
<!ELEMENT packages (package*)>
<!ELEMENT package  (name,md5,url)>
<!ELEMENT name     (#CDATA)>
<!ELEMENT md5      (#CDATA)>
<!ELEMENT url      (#CDATA)>
]>

Another way would be a database, mysql, or sqlite, but I don't see the point
because with an xml file it's easier to store in a server, and we don't need
the speed that provides a database engine.

Programming technology:

I suggest Python, since there's a really easy way to parse html files,
handle XML files,
compute md5, upload the tarball to another server if necessary, and
download the tarball
from the upstream server.
Since we don't need too much speed to do this, the python interpreter
low speed
doesn't matter.

Application workflow:
1)
$ ./app [--add,-a]<ENTER>
Package name: <write name><ENTER>
Not a valid name, please try again... [if the name contains spaces, or
rare chars, and display the "Package name" prompt again]
Package URL: <write url><ENTER>
Not a valid URL, please try again... [if the URL contains spaces, or
rare chars, and display the "Package URL" prompt again]
Downloading package source...
Calculating md5...
Storing data...

2)
$ ./app [--del,-d] [--filter,-f package]<ENTER>
1) clisp
2) cmucl
3) cl-smtp
...
N) cl-something

Which package do you want to delete? Please write the package number
from the list:
<write package number><ENTER>
The package number is doesn't correspond with any package from the list
displayed.
Please try again... [if the number is greater than N or less than 1 and
display the prompt again]

3)
$ ./app [--check,-c] [--package,-p package]<ENTER>
[if the -p option was provided]
Checking for updates of package
There is a new version of package, the tarball is saved in the "new"
directory.
[else]
Checking for updates...
........[may be display a dot for each package checked, or display a
more verbose output]

4)
$ ./app [--update,-u package]
Please type the new version: <write new version><ENTER>
Not a valid version, please try again... [if it's not a valid Gentoo
version, and display the prompt again]
Mirroring the tarball with name package-newversion.tar.gz in mirrors
directory...

Where to place this:

My first idea was to place this in woodpecker, and the tarballs in my
www directory. May be
there's a better, more collaborative way of doing this. May be the infra
guys have something
for things just like this. Mirroring the tarballs in another server is
really easy through ftp with
python, so we could run this on woodpecker and mirror the tarballs
somewhere else.

****** End ******

So I think that's about it.

While we are developing the mirroring system and moving the packages, we
will RFC the common-lisp eclass needed from the overlay in gentoo-dev
ml. Once this eclass is in the main tree, and we have the mirroring app,
the real porting can begin.

What do you think?

Cheers,
Tomás