[gentoo-soc] R_Overlay: Automated overlay maintenance - final report

[André Erdmann]

Hi everyone,

== Overview ==

This project is about creating an overlay of R packages, without
requiring much interaction. Overlay creation itself was already
implemented in last year's GSoC project "Automatically generated overlay
of R packages", with "roverlay" [0] as result. This year's work focused
on extending roverlay and facilitating overlay maintenance.

I'm going to give an insight on what's been done this summer, without
going too much into details.


== Overlay creation features ==

* RSUGGESTS USE_EXPAND variable

Instead of a single "r_suggests" USE flag, overlay creation now produces
per-package USE_EXPAND flags (r_suggests_<sth>). These flags can also be
renamed and grouped.

* extended package rules system

package rules match a set of packages by name, origin,... using boolean
expressions and apply actions to them, which manipulates their metadata
or causes packages to be ignored. The general concept and syntax/parser
was implemented prior to this year's GSoC, but lacked actions. Now,
packages can be "relocated" (category, name, $SRC_URI), dependency
resolution can be influenced (add/ignore dependencies) and ebuild
variables (LICENSE,...) can be set. Additionally, support for
"else"-blocks has been added, which makes writing of rule files easier.

* support for SLOTs in resolved dependencies (including sub-slots)

* patch generated ebuilds, import ebuilds into the overlay


== Improved overlay quality ==

* selfdep validation

This feature ensures that all dependencies on packages provided by the
generated overlay are satisfiable ("candidates" exist), which improves
the overlay's consistency.

* dependency resolution: look up package dependencies in own repo first

More accurate dependencies if a package is available from more than one
repo.

* handle file collisions (package available from >1 repo)

Very important, because it keeps the package mirror directory
collision-free.

* revbump if upstream modifies a tarball without renaming it


== Misc ==

* (dependency resolution) console reimplementation with support for tab
completion, command history

* API for using roverlay's functionality externally (dependency
resolution and remote-sync)

* "Event Hooks", execution of scripts on certain events, e.g. automatic
metadata cache creation (suggested by heroxbd) and git commit/push after
overlay creation

* up-to-date documentation [1]


== Helper scripts ==

* roverlay-setup

This script takes care of generating a config file, creating directories
with proper permissions and managing event hook scripts.
It's mostly intended for setting up a roverlay server, which can be done
with "roverlay-setup" or "roverlay-setup-interactive"/"emerge --config
roverlay".

* roverlay-sh, a /bin/sh wrapper which sets up a environment similar to
what's used for "Event Hooks" (access to variables like $OVERLAY)

* roverlay-status

Creates status reports based on mako templates and roverlay's stats. Can
also be used as CGI script.


== Conclusion and Future directions ==

After 2 summers and some work in between, roverlay is now (more or less)
complete. What's left to do is some more testing, updating the roverlay
server and then making the R overlay available via layman.

Finally, I'd like to thank Denis (Calchan), my mentor, for suggestions,
testing and guidance throughout the past months, and for encouraging me
to participate this year again ;)

Credits also go to Benda (heroxbd) for a patch to the R-packages eclass
and suggesting metadata cache creation (which then led to the "Event
Hooks" feature), and Ebo for making me aware of a dependency resolution
issue.


[0] http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=summary
[1]
http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=blob_plain;f=doc/html/usage.html;hb=HEAD

Regards,
André E.