[gentoo-soc] Fwd: [soc proposal] Automatically generated overlay of R packages

[André Erdmann]

Hi,

I've just submitted my proposal (included in this mail) for this
year's gsoc project idea 'Automatically generated overlay of R
packages'. Feel free to comment and criticize, it helps me a lot to
improve it.

Kind Regards,
Andre Erdmann

<<<
--- Table of contents ---
1 Abstract
2 Objective
3 Implementation ideas
 3.1 General
 3.1.1 Distributing and maintaining the overlay
 3.1.2 Package mirror
 3.1.3 Scripts

 3.2 Getting packages & information from CRAN, Bioconductor
 3.2.1 Fixing package metadata
 3.2.2 Getting packages from BIOC
 3.2.3 Getting packages from CRAN

 3.3 Satisfying dependencies
 3.3.1 R package deps
 3.3.2 system deps
 3.3.3 optional deps (sys or R-pkg)
 3.3.4 Additional thoughts

 3.4 Generating ebuilds

4 Deliverables
 4.1 Final (August 20)
 4.2 Mid-Term (July 9)

5 Timeline
6 Biography / About me
7 Extra information
 7.1 Use the tools that you will use in your project to make changes to code
 7.2 Participate in our development community
 7.3 Contact info
 7.4 Working hours

--- Table of contents ---

1 Abstract
---
The aim of this project is to create an overlay that contains
automatically  generated ebuilds for R packages from CRAN and BOIC.

2 Objective
---
Creating an ebuild for an R package means
* check for newer versions
* discover all dependencies (system deps, too)
* fix package metadata if required
* upload the R package to a mirror
* write the ebuild and create/update the Manifest file.
It's impossible to do that for all R packages from CRAN/BOIC due to
the high number of available packages. Using scripts to automate the
process addresses this issue and can be used to create an overlay.

Clean code, proper logging and good documentation are key components
of this project since integration into the Gentoo infrastructure is
intended.


3 Implementation ideas
---
3.1 General

3.1.1 Distributing and maintaining the overlay

Using git offers easy updates/rollbacks via git push/pull/checkout/...
and would allow to separate overlay creation/update from overlay
hosting.

the abstract overlay layout would be:
 /dev-R/
 /dev-R/<origin>-<package_name>/
 /dev-R/<origin>-<package_name>/<origin>-<package-name>-<${PVR}>.ebuild
 /dev-R/<origin>-<package_name>/{[metadata.xml,],ChangeLog,Manifest}
 /eclass/
 /eclass/<files that help, e.g. r_overlay-fix-metadata.eclass>
 /profiles/...

example: /dev-R/CRAN-numDeriv/CRAN-numDeriv-2012.3.1.ebuild

3.1.2 Package mirror

To be figured out.
Probably nothing special, could be some directory/file tree structure like
 "<r-overlay_distfiles-root>/dev-R/<portage-atom_base>/<portage-atom_base>-<${PVR}>.tar.gz"

3.1.3 Scripts

Will be implemented using bash or perl and kept/developed in an extra git repo.
An ebuild that installs the whole script set will be part of the gsoc
project, which means that an overlay client could be used to maintain
the overlay (with proper write/push permissions on the server).
There will be an ebuild-generator, src_package-to-mirror uploader
(rsync or mv/cp, ...), a cran/boic importer ("syncer"), test scripts,
a dependency resolution map etc..
Logging will be done in a machine/parse-friendly way and a converter
into a more human-readable format will be supplied, too.

3.2 Getting packages & information from CRAN, Bioconductor
    (and possibly Omegahat etc. but this is not part of the gsoc project)

The DESCRIPTION file contains all(?) information that is required to
generate an ebuild for a package.

3.2.1 Fixing package metadata

Package metadata could be fixed while emerging (using an eclass) or at
overlay creation/update-time (which requires gentoo-specific source
tarballs).
Further discussion has to be made in order to choose the right option here;
I would choose the former one and add flags for a metadata-fixer
eclass ("E_ROVERLAY_FIXMETADATA='this that etc'") to the generated
ebuild, leaving the package tarballs unchanged.

3.2.2 Getting packages from BIOC

BIOC offers all source packages in a tree structure (<pkg_name/...>)
via svn, making an update check as easy as 'svn status --show-updates'
(or similar).
The DESCRIPTION file can be parsed directly; tarball creation is
necessary for the distfiles mirror, so no specific pros/cons regarding
3.2.1.

3.2.3 Getting packages from CRAN

CRAN source packages can be mirrored as tarballs (containing the
package files) using rsync. Each tarball has to be extracted (at least
partially, see 3.2.1).


3.3 Satisfying dependencies

Relevant fields in the DESCRIPTION file are 'Depends, [Suggests],
[SystemRequirements]', maybe more ('[Enhances]',...).

3.3.1 R package deps

a Create all ebuilds blindly (don't check these deps) and perform
   a consistency check ("can satisfy") afterwards
b Queue the creation of an ebuild E until all of it's dependencies have been
   added (or no package left to add which would imply dependency failure)
b.1 Could try to satisfy remaining dependencies as sys deps
     (would do it for "R" in 'Depends: ...' anyway).
=> would choose b here (would also allow 'stable' git commits per
ebuild/R package).

3.3.2 system deps

The task is to find dependency aequivalents in the portage tree, using


* a static dependency alias map (as file,...)
   "<this dependency>" means "<atom-base>",
     e.g. "gnuplot" => "sci-visualization/gnuplot"


  May be overridden by a per-Rpackage alias map since different dependencies
  could have the same name (varies by package author or
  even within portage - app-misc/screen vs app-vim/screen (just as an example))

* a dynamic dependency alias map ("satisfied before, but not confirmed")
   Created by...

* educated guessing
   by name; or try using string metrics (case sensitivity, Levenshtein, ...))

Proper logging of the dependency resolution is a crucial part here.


3.3.3 optional deps (sys or R-pkg)

'Suggests:' could be controlled via USE-Flags (IUSE+=" <short name of
optional dep> <another one>")
'Enhances:' could be ignored or added as USE-Flags, too.

3.3.4 Additional thoughts

A simple web ui or script (for ssh or local usage) could be implemented to
ease such things (dependency mapping, maybe more; not part of the gsoc project
for now, but kept in mind as a nice-to-have feature).

3.4 Generating ebuilds

Ebuilds use generic eclasses that handle most of the installation process.
Extra per-package dependencies could be 'injected' where required
(using an extra dependency map,...). If a R package can be installed
from both CRAN and BOIC, choose the 'best' one (to be clarified) or
add both (and block each other in the ebuilds).


4 Deliverables
---
4.1 Final (August 20)

scripts : cran/boic importer,
           ebuild generator with dependency resolution,
           src_package-to-mirror uploader,
           log reader,
           consistency checkers etc. (see 3.1.3)
           ebuild for the overlay generation script set

documentation : man pages for the scripts,
                       implementation documentation ("what the system
does and how")
                        as wiki or GuideXML page (needs to be discussed)

overlay : functioning; filled with R package ebuilds, eclasses, etc.

4.2 Mid-Term (July 9)

scripts             : ebuild generator
documentation : implementation documentation 70-80% done
overlay            : eclasses


5 Timeline
---

*  - May 1 (Tu): read documentation, concretize the implementation ideas

* May 2 - May 21 Mo): write the (a priori) project documentation of
what the system does and how

* May 22 - : begin to discuss with the Gentoo infrastructure team
whether they would agree to install the overlay system on Gentoo
infrastructure

* May 24 - June 3 (Su): write eclasses and the ebuild generator
without dependency resolution

* June 4 - July 1 (Su): add dependency resolution to the ebuild generator

* July 2 - July 9 (Mo): prepare for the Mid-term evaluations; improve
documentation

* July 10 - July 30 (Mo): sync scripts (import from CRAN / BOIC, push
R package source to mirror), test scripts

* July 31 - August 10 (Fr): write man pages, the log reader and an
ebuild for the overlay generation script set

* August 11 - August 20 (Mo): improve documentation



6 Biography / About me
---
I'm a twenty-year-old undergraduate student from Stuttgart, Germany.
My major subject is Computer Science in which I'll get my bachelor's
degree in 2013 or 2014.
I like to tinker with Linux distributions and computer devices
(Dreamplug, mips starter kit - just to name a few). I'm not afraid of
writing   programs/scripts  whenever required (or supposed to be
helpful/..). I've seen and used several programming languages
(including different paradigms) such as C/C++, Haskell, Prolog,
Python, Perl, mips-Assembler (spim), Ada, Java and Shell
(bash/dash/hush) which gives me a basic overview on how to implement
things.

I've been using Gentoo for 3 years now and started a small
gentoo-related project last year. It's main effort is called
tlp-gentoo [1], which automatically converts TLP's source code (a set
of scripts that implement power management) into gentoo-usable one
using regex, patches etc..
Ebuilds for the converted source are provided in an overlay (tlp-portage [2]).
Moreover, I'm doing code reviews for the upstream project on a regular basis.
I've chosen this project 'cause I want to go further than that.
An automatically generated overlay is a thing I've never done before
and I'm really interested in realizing that. In addition, I'll be
using a few R packages in near future, but my motivation is the
process of overlay creation. I'm also referring to the tlp project to
show you that I don't plan to bail out after the initial work has been
done.


7 Extra information
---
7.1 Use the tools that you will use in your project to make changes to code

fixed bugs at bugs.gentoo.org : none so far; to be done.

7.2 Participate in our development community

You can find a mailing list entry from me <in this mail>.

7.3 Contact info

email  : dywi at mailerd.de
irc      : dywi at irc.freenode.net

home mailing address: <removed>

phone number: <removed>

7.4 Working hours

Mo - Sa, 8 am - 9 pm UTC
actual working time may vary, but sums up to 30-40 hours per week.


---
[1] https://github.com/dywisor/tlp-gentoo
[2] https://github.com/dywisor/tlp-portage
---

>>>