[gentoo-science] sage, gentoo-prefix & lmonade was: log messages

[Burcin Erocal]

Hi Keshav,

thanks for pointing out this thread. "log messages" is an interesting
title to discuss the future directions for Sage. :) I added
gentoo-science to the CC list, since that is the most likely location
for sage-on-gentoo discussion and the gentoo-developers might be
interested in the contents of this thread.

On Mon, 20 Feb 2012 22:45:24 +0800
Keshav Kini <keshav.kini@×××××.com> wrote:

> Robert Bradshaw <robertwb@×××××××××××××××.edu> writes:
> > On Fri, Feb 17, 2012 at 1:09 AM, Keshav Kini
> > <keshav.kini@×××××.com> wrote:
> >> Robert Bradshaw <robertwb@×××××××××××××××.edu> writes:
> >> In the category of "glue code" I meant to include everything in
> >> $SAGE_LOCAL/bin/sage-*. I see much of that stuff as more related to
> >> maintenance of the entire distribution of software we ship than to
> >> the actual Python/Cython code in the Sage library. Of course, if
> >> we do switch to Prefix or something like it, most of it will become
> >> unnecessary, or can actually be fit into the portage data tree as
> >> "Tools", so I see your point.
> >
> > Yeah, tools describes it well (or at least what remains after the
> > package management stuff is gone, like testing and coverage and
> > stuff).
>
> Well, I meant that there is a "Tools" directory in sage-on-gentoo. I'm
> not sure if this is a standard thing in the directory structure of a
> Gentoo Prefix tree, or, if it is, what sort of stuff is supposed to go
> there, but I see some utility scripts, presumably written by François
> or Christopher, in there and was suggesting we might be putting our
> sage-* scripts there too. But I don't know enough about Prefix to be
> sure whether that makes sense.

AFAIK, Tools contains the scripts that are used to maintain the
overlay. sage-on-gentoo contains these packages for Sage:

  sage-baselayout
  sage-clib
  sage-data-conway_polynomials
  sage-data-elliptic_curves
  sage-data-graphs
  sage-data-polytopes_db
  sage-doc
  sage-extcode
  sage-flasknotebook
  sage-notebook
  sage

baselayout is what gentoo calls basic scripts. That is the package
containing the relevant part of sage-scripts. While I like the
separation of the data packages (I assume these don't depend on code
from the sage library), I am not sure about the clib, doc and the
library separation. I'd like to keep those together to match the Sage
development pattern as close as possible.


> >>> I think Sage will be monolithic and Windows be VM for the near
> >>> future at least, with a larger percentage of people using a Sage
> >>> install "in the cloud" on a university or otherwise hosted server
> >>> for the near-mid term future. But as you said who knows...
> >>
> >> Yes, I agree. That seems like the most likely future, at the
> >> moment. However, William has asked me to write a Sage Enhancement
> >> Proposal for switching to git, and I think we concluded on IRC
> >> that it might make sense to make a long term timeline for other
> >> big changes as well, such as what we're talking about now - or at
> >> least a proposal for such a timeline :)
> >
> > +1, which is why we should be having this discussion now. Switching
> > to git could probably happen a lot faster than revamping the build
> > system, but at least we could consolidate some of the repositories
> > and re-seat it at a higher level in the directory structure so
> > moving to a new build system is "just a patch" :).
>
> Absolutely. Switching to git is much easier than revamping the build
> system. The only reason I'm mentally grouping them together is that
> consolidating SPKGs, which would be part of remaking the build system
> as I've described, is also related to version control.
>
> But that actually doesn't make a lot of sense now that I think about
> it. We would basically be throwing away the old SPKGs' files, so I
> don't know if it's even really worth preserving their history in the
> portage tree of Prefix, assuming we move to Prefix - we could easily
> preserve their history somewhere else instead.
>
> Of course, even if it's not really related to the switch to git, I'd
> definitely like to keep a build system revamp on the horizon!
>
> > Thanks for filling me in. I was aware of the sage-on-gentoo effort,
> > but never looked into it technically. The more I see of it the
> > better it sounds. This seems to have precisely the flexibility we
> > need.
>
> I'm very glad to hear you think so too! Spread the word! :P
>
> Just to be clear, sage-on-gentoo itself has no goal of being a way to
> distribute Sage in general. It is simply a port of Sage and its SPKGs
> to the Gentoo package management system, using the "overlay"
> architecture the Gentoo distro has (similar to Personal Package
> Archives on Ubuntu, if you're familiar with those). Basically you
> just put something in a config file on your Gentoo system which says
> "augment the package repository with ebuilds from sage-on-gentoo",
> and you can now install Sage system-wide.
>
> It is Burcin's lmonade project which combines sage-on-gentoo with
> Gentoo Prefix, which is a portable version of the Gentoo package
> management system which can be run in a folder on any system, and
> more importantly, can be customized to supply any set of packages,
> not necessarily the entire set of officially sanctioned packages that
> comprise the Gentoo Linux distribution. As the name suggests, it's
> similar to how you might use `./configure --prefix=/some/dir` in
> order to install a program to /some/dir/bin rather than /usr/bin, say.

Thanks Keshav, that's a great summary of the relation between
gentoo-prefix, sage-on-gentoo and lmonade [1]. For the record, I'd be
very happy to see Sage use lmonade for the distribution aspects.

[1] http://www.lmona.de/

lmonade was born because I thought that a customized Sage distribution
like William's psage [2] was not maintainable in the long term. It
still needs some work to cover all functionality of the Sage
distribution, such as relocation and support for development of Sage.
Given time, these are not hard problems. Unfortunately, I haven't had
much time to work on lmonade lately.

[2] http://purple.sagemath.org/

> Obviously, this is pretty necessary for Sage proper, which likes to
> sit in its own folder and have control over its own private copy of
> various things. To do otherwise would be to abandon doing our own
> packaging altogether, which I think is basically impossible. So if we
> were to adapt anything, it would be lmonade, not pure sage-on-gentoo.
>
> > One thing seems a bit unclear: are the sage-specific changes pushed
> > into some global repository for, e.g. gap, or are they local to the
> > sage install? If they are, how is it proposed that versioning be
> > handled (will there be a sage-5.0 and sage.5.1 tag, or sage-r1 and
> > sage-r2, ?) Or is this repository of Sage use metadata something
> > that we maintain?
>
> I'm not really sure what you mean by a "global repository" or "local
> to the sage install". Versioning will be basically the same as how we
> do SPKGs right now, namely that we'll take the upstream version of the
> package and add on a "Sage patch version", which in portage must be
> represented as a suffix of the form -r1, -r2, -r3, etc. So for example
> what is now sympow-1.018.1.p11.spkg would become, in the new system, a
> possible set of patches, plus sympow-1.018.1-r11.ebuild, containing
> (among other things) a URL from which to download the sympow 1.018.1
> source tarball.
>
> Here is a highly abbreviated directory tree of lmonade, as it appears
> in the installation I made the other day, for reference:
>
>     lmonade/
>     | dist                  package management / build system
>     | | distfiles/          downloaded upstream source tarballs
>     | | log/                build logs
>     | ` portage/            ebuilds, patches, etc.
>     |   ` sci-mathematics/
>     |     ` gap/
>     |       | files/
>     |       | | gap-4.4.12-sage-and-steve-lintons-itanium.patch
>     |       | ` gap-4.4.12-sage-strict_aliasing.patch
>     |       | gap-4.4.12-r2.ebuild
>     |       ` metadata.xml
>     ` local/                software is installed here
>       | bin/, etc/, include/, lib/, libexec/, sbin/
>       ` share/
>         ` sage/
>           ` devel/          corresponds to $SAGE_ROOT/devel today
>
> So the "package repository" would be dist/portage, and the "sage
> library repository" would be kept at local/share/sage/devel/sage-main
> (or hopefully just ...devel/sage, once we get rid of `sage -clone`).
> I don't actually know my way around this directory structure very
> well, but those are the main takeaways, I think.

Your description is accurate. The development repositories will be in
devel/. pynac and pypolymake live ebuilds already put their files in
this location. The separation of the sage ebuilds stopped me from using
the same scheme for sage up to now. The goal is to put the development
repository for a package foo in devel/foo once the user runs

lmnd devel foo

> You can see the contents of gap-4.4.12-r2.ebuild here:
> https://github.com/cschwan/sage-on-gentoo/blob/83deaae5/sci-mathematics/gap/gap-4.4.12-r2.ebuild
>
> There are some seeming oddities there, like the USE flag "emacs" which
> pulls in Emacs as a dependency and then installs the Emacs mode for
> GAP into it - clearly not something we want to do, and indeed it's not
> enabled by default in lmonade. It's there because, as I said,
> sage-on-gentoo is meant to allow you to install Sage into a full
> Gentoo system, where it *would* make sense to do that.
>
> I guess to avoid extra maintenance load we'll just have to leave that
> extra stuff in, both in sage-on-gentoo and on our own packaging repo.
> (And who knows, maybe someone *does* want a local copy of Emacs or
> whateverinside their Sage installation...) I doubt we'll be able to
> actually use sage-on-gentoo directly as our packaging repo, but at
> least we can keep them similar (?).
>
> By the way, it looks like only three packages in sage-on-gentoo
> actually have a "sage" USE flag defined (PolyBoRi, Python, and
> LinBox). It seems that the others, including GAP, just install Sage's
> patches whether you want them or not. I'm not sure why this is -
> maybe it's just because it's painful to specially create a "sage" USE
> flag for every package, or maybe it's because sage-on-gentoo wants to
> get Sage's patches included in Gentoo proper as default patches, or
> maybe there's some other reason.

AFAIR, the only patch contained in the GAP spkg was a fix for itanium
from Steve Linton. This is useful in general. No need to hide it behind
a use flag. 

> I'm getting way out of my depth here, as you can tell, so I'm CCing
> Burcin and François, in case they haven't been reading this exchange
> and have something to add or remove from what I've been saying! I'm
> sure Burcin in particular has thought a lot more about where all this
> could go than I have. Of course, feel free not to jump in if you're
> too busy.

Clearly you've also thought about this a lot. We seem to agree on almost
everything. I am also happy to see that even with the current lack of
documentation, main design criteria come through. I'd really like to
see some of your explanations on the lmonade wiki actually. :)

> Well, actually I'm going to send this message through Gmane, so maybe
> they won't be CC'd after all. We'll see :)

Gmane CC works.
 
> >> I think we can get a pretty good alternative to this with the split
> >> system I'm describing (nimble package management repo for everyone
> >> and carefully reviewed sage repo for creating releases from), if I
> >> understand correctly. We would include the ebuild for Sage itself
> >> inside the Sage repository, and export it to our package
> >> repository when we make a release of Sage.
> >
> > By "our package repository" are you thinking of something maintained
> > by us, or by gentoo?
>
> Gentoo's involvement with sage-on-gentoo and lmonade ends at providing
> the package manager software, and in the case of sage-on-gentoo,
> putting sage-on-gentoo on the official list of overlays, so that you
> can attach sage-on-gentoo to your Gentoo system with a simple
> command, `layman -a sage-on-gentoo`.
>
> So our package repository would be maintained by us. Of course, we
> would only need to actively maintain the ebuilds and patches for
> programs which we actually patch. No more would it be necessary to
> create an SPKG for a program which needs no Sage-specific patches, as
> long as that package is already available on Gentoo - we can just
> steal the ebuild file from Gentoo's portage tree.

We could also share the ebuilds with Gentoo of course. Most Gentoo users
don't mind using bleeding edge versions of software anyway. The whole
point of using portage instead of the spkg system is to eliminate the
need to duplicate this packaging work.

> > But yes, this sounds like a great idea! So the
> > collection of .ebuild files (and their supporting patches, etc.) is
> > in the Sage repository, but periodically exported somewhere central
> > (owned by us? Gentoo?) that people pull from, right?
>
> Er, nope. The ebuild corresponding to the Sage library itself would be
> stored in the Sage repository, and periodically (upon the release of a
> new Sage version) be exported into our package repository. All other
> ebuilds and patches would be put directly into that package
> repository, which would be hosted live somewhere, say on github. Our
> package repository would look very similar to the current
> sage-on-gentoo repository, i.e. be a portage tree providing ebuilds.
> The other repository would be the Sage repository.
>
> In other words, we would consider the Sage library as another
> "upstream package", except that it's aware of its part to play in the
> Sage package repository, and thus provides its own ebuild as part of
> the distributed source code each time it distributes a new version
> (i.e when we release a new version), whereas for other packages we
> would have to write our own ebuilds for them (or borrow them from
> Gentoo), not constantly, but whenever the upstream developers
> released a new version.

I don't see the need to keep the Sage ebuild in the repository
containing the Sage library. It should also be in the package
repository. Just like:

https://bitbucket.org/burcin/lmnd-prefix/src/tip/sci-mathematics/sage/

(I really need to update. Actually, I was just working on using git to
automate synchronizing with gentoo-x86, gentoo-prefix, gentoo-science
and sage-on-gentoo.)

> > All I could ask for more is that the package management/build system
> > be functional (hermetic) and reversible.
>
> Well, I don't know how hermetic portage is, but it's a sight more so
> than SPKGs, anyway :)
>
> I remember someone was discussing using Nix_ for Sage on sage-devel
> several months ago. But I don't think it's really mature enough to do
> so, not to mention that we lose the advantage of being able to steal
> packages from a mature Linux distribution like Gentoo.
>
> .. _Nix: http://nixos.org/nix/

Dag mentioned it a while ago:

https://github.com/dagss/scidist/blob/master/ideas.rst

Being able to reuse the work of gentoo-prefix, gentoo-science and
sage-on-gentoo folk was exactly my reason for going with gentoo-prefix
and portage. Many thanks to the Gentoo developers, Francois and
Christopher for making this possible.



Cheers,
Burcin